Amsterdam and Germany 9 Day Itinerary

Sounds to me like a nice amount of time in some nice places.

"...hoping for a few small, pretty towns for this trip. I have been looking at Rüdesheim, Wurzberg, Bamberg and Munich..."

The last 3 places aren't small towns in the way Rüdesheim is. Würzburg and Bamberg are each roughly ten times larger. Both are nice places with UNESCO World Heritage recognition, however, with an ample number of things to see and do, and Würzburg is a good hub for day trips by train (stay there, day trip to Bamberg and elsewhere.)

"We both enjoy wine, beer..." Train travel will make getting around that much safer. Würzburg and R'heim are wine towns that are surrounded by other wine towns; Bamberg is a beer mecca.

"We also like the idea of Rothenburg but not a huge fan of the fact it is crowded with tourists."

Rüdesheim suffers a bit from "Rothenburg syndrome" (heavy influx of tourists and tour buses in summer.) I'd suggest spending a few hours there but staying elsewhere, maybe in a more central Rhine town like St. Goar, which is by no means tourist-free but is a very nice place and would make day trips to the other Rhine (and Mosel) towns a little easier (it has a ferry crossing with train access across the river, unlike some other towns.)

Nice small towns and sights you could visit from a St. Goar base:

Boppard, Oberwesel, Bacharach, Bingen (to catch a cruise boat,) Marksburg Castle in Braubach, Rüdesheim, Burg Eltz Castle (near Moselkern,) Cochem, Remagen, Linz

(The Rhine and Mosel rivers make for very scenic train rides. btw.)

And from a Würzburg base:

Marktbreit, Ochsenfurt, Sommerhausen, Veitshöchheim, Bad Windsheim (open-air museum,) Iphofen (in German but see photos,) and of course Bamberg and Nuremberg.

"I assume we need to stay in Frankfurt Thursday..."

If you prefer, there are DIRECT trains from Würzburg that take 1.5 hours to reach the FRA airport station. Board the 6:26 in Würzburg, arrive 7:55 at FRA.

Rüdesheim is very touristy, but it has one unique feature that I think justifies a visit. On weekend nights, many of the restaurants have live Schlager music performances. You usually encounter several bachelorette or stag parties from the surrounding cities, and it makes for a fun and very lively atmosphere. Schlager music may not be to everyone's liking, but it will definately provide you with a travel memory. Definately consider some time on the Mittelrhein.

Rothenburg is nice, but not unique enough to justify a long detour.

As noted by Russ, Bamberg and Würzburg are decent sized cities, not "small towns". But they are worth a visit.