It's good that you have May 4,5, and 6 for seeing Berlin, as well as several hours on May 3 (assuming mid-day arrival.)
I think that after Mannheim, it's probably a mistake to attempt so much of Bavaria with just 4 days. I would eliminate Munich, Regensburg and Neuschwanstein, all of which add to your train travel time and expense; even the Bavaria that surrounds Nuremberg is far more than you can handle in that short time. You'll spend half a day May 2) in transit to Nuremberg to start with. So with 3.5 days of sightseeing you will still face hard choices. Nuremberg itself could occupy 2 days. Nuremberg is also a good travel hub... Bamberg (45 min.) and Rothenburg (1.25 hrs) are only two of the potential day trip options... there's also Bad Windsheim (55 min.) and its excellent open-air museum, the town of Bayreuth (55 min.) and its opulent opera house (UNESCO World Heritage status,) the Main River city of Würzburg (1.25 hrs,) and the tiny and adorable walled town of artists and winemakers, Iphofen (45 min.) A day trip from Nuremberg into Munich for a few sights there would be possible as well.
There are those who might disagree, but I think the broader Nuremberg area offers much more variety and an altogether superior experience for first-timers to Germany than Berlin does. I could easily see April 29 - May 4 in your Nuremberg base. Train to Berlin to fly out. (Nuremberg has an airport as well, if you aren't already locked into Berlin; access via the Nuremberg subway system is easy.)
As for castles... Nuremberg has one, and you may have a chance to visit one or two from your Mannheim base. N'stein would be an extraordinary journey away from your route, considering it offers but a 30-minute tour (and it's not a genuine castle, but a late 19th-century residential palace.)