While living near Nuremberg I took various friends around and their interests determined where we went and how long we spent. Plus a good "Drei-im-Wechla" lunch, of course, (although I have no idea now which of our favorite restaurants will survive). The local transit is very good S-bahn (surface trams) and U-Bahn (subway trains), which shortens travel between destinations.
4 (3) nights gives you three (two) full days so one site outside the center per day (Documentation Center, Nuremberger Process, etc. ) and one or two sites (Art Bunker, Castle, Toy Museum, Albrecht Durer House, GNM, etc) within the city will work well.
Check out their museum guide:
https://museums.nuernberg.de/nuremberg-municipal-museums/
The Documentation Center is worth a morning or afternoon, but is a 15 minute S-bahn ride SE of the Hauptbahnhof. The "Rally Grounds" as noted above aren't much and are mostly used these days for rock concerts (or have been). The Germanisches National Museum is wonderful (the oldest globe, great pre-historic, Medieval, Renaissance exhibits, excellent art, etc) and is near the train station (but inside the city wall). GNM is worth a half day at least, or probably half a day on each of two days for some variety time in between. The Art Bunker is up near the castle on the north side of town and a half hour walk from the GNM. Nice walk through the main market square (home of the Christmas Market) with its famous fountain, and various churches Lorenz Kirche, Frauenkirche, St Sebaldis Kirche (which is now a war memorial). The castle is nice castle (totally rebuilt of course after the US Army Air Force did it's thing in 1945) and may deserve a walk through but not much to really see. In June they have beer festival in its moat.
You don't mention the Nuremberger Process - which is the actual courtroom where the Nuremberg Trials were conducted. It is out the U-bahn several stops west of the main station. Because the Germans wanted/needed it as a normal courtroom it was converted back into everyday use and can only be visited on weekends when the court is not in session - but it has an excellent museum about the trials in the upper floors of the building, which used to be open most days.
Just a note: Both the Documentation Center and the Nuremberger Processes courthouse leave most visitors rather shaken.... it is not easy to contemplate the horrors .... so something lighter to follow is usually advisable.