we would like to try to find some smaller towns for a bit of a respite
I'm curious about what you want to get out of a small town. BTW, at 73,000, I would not consider Bamberg a small town - maybe smaller than the cities you've been in, but not really small. (The median size of towns I've spent a night in in Germany in the last 20 years is less than 8,000 people.
So, is it just that you want to stay in a town, with some attractions, that is smaller than the cities you've been in, or do you just want to visit a small town, one completely isolated from tourism, and see how ordinary Germans live.
If the latter, I would suggest a night or two in Lohr am Main. Lohr is what I would call a perfect honeymoon venue - just a pretty place with nothing really going on (but honeymooners make their own entertainment). At 16,000 population, it is really bigger than my typical small town, but the major source of employment there is a large Bosch factory, which inflates the population some. It's just a fun place to relax. There is a main street with a lot of fachwerk buildings and plenty of shops and restaurants. There is one museum in town, in the old palace, dedicated to the story of Snow White, which local legend says originated here.
Lohr is on the main train line between Würzburg and Frankfurt and about a half hour's train ride from Würzburg, so a day trip there would be possible, as would be a day trip to Bamberg or Rothenburg.
I can tell you more about it if that's what you mean by a smaller town.