When you say "Bavaria" you are referring just to the Bavarian Alps, I think.
In the map below, the Bavarian Alps are at the south end and tan in color. Everything green to the north is "Bavaria" too.
Map of Bavaria
Bavaria isn't exactly chock-full of castles. They tend to be fairly few and far between.
"But the rest of the time we are hoping for castles and rural German experiences."
Your best bet for rural and castles is closer to Berlin than the Alps and the Black Forest... The Upper Middle Rhine Valley and the Mosel River Valley are close to each other, bathed in castles, and... very rural, and beautiful. Both places happen to be on the way to the Black Forest, and both offer hiking, biking, river cruising, wine tasting, and other nice travel options.
Map with Rhine Castles between Koblenz and Bingen... click on castle icons for info.
Map showing location of the Rhine and Mosel Valleys and the Black forest in Germany
And between the Rhine and the Black Forest you will run into Stuttgart, home to both Mercedes Benz and Porsche.
The Black Forest itself isn't filled with tall mountains but still qualifies as one of Germany's most scenic rural spots. I've been there myself on at least 6 different multi-day stays, exploring new places each time. I've read Rick's Black Forest suggestions and seen them, and they are indeed somewhat underwhelming... Baden-Baden, Freiburg, Triberg, and Staufen were some of my least favorite places. Makes you think he may not have gotten around to some of the nicer places. Have a look at some others... Gengenbach, Gutach im Schwarzwald (Open Air Museum), Schiltach, St. Blasien, Haslach, Calw, Titisee and Neustadt (for the scenic hiking, mostly) and Löffingen (Wutachschlucht hiking)
Railway map for these last three:
https://de.academic.ru/pictures/dewiki/86/Verlaufskarte_Hollentalbahn_2.png
I like the Black Forest also because it has undergone less in the way of modern development than the rest of Germany, as a rule. Old ways have persisted a bit more than elsewhere.
I think you'll really feel like you're in Germany with a Berlin > Rhine/Mosel > Stuttgart > Black Forest itinerary.