Small town Germany

Cologne lies in a largely non-scenic section of the Rhine River, but not too far to the south, roughly between Bonn and Bingen, is the Middle Rhine Valley (the yellow section of river on the linked map.) There you'll find great natural scenery, a big number of attractive old-world towns, and very good train connections on both sides of the Rhine.

PART of the Middle Rhine Valley - the Upper Middle Rhine Valley between Koblenz and Bingen - has been designated a UNESCO World Heritage site. You'll find information on this smaller area here.

The smaller towns of interest in the MRV are...

Remagen
Linz am Rhein
Braubach (home of Marksburg Castle)
Boppard
St. Goar (home of Rheinfels Castle
Oberwesel
Bacharach
Rüdesheim

Any of these might make a good base town for you.

Koblenz, not a small town, is generally a less attractive destination, although it does have a fortress (Ehrenbreitstein) which might make for interesting sightseeing.

At Koblenz, rail lines up the Mosel Valley and the Lahn River Valley offer access to additional small towns of interest.

Mosel: look into Cochem, Traben-Trarbach, Beilstein, Bernkastel, Trier
Lahn: look into Bad Ems, Diez (castle/hostel) and Limburg

Area rail map:
https://www.vrminfo.de/fileadmin/data/pdf/2017/RLP-Ticket_streckennetz.pdf

Not 100% sure what your plan is, but note that most train lines are radial. That means you'll start every daytrip day with a run into the HBF of Cologne or Munich, and then a ride to your actual destination. You may (?) need a local bus from your "town" stay to the SB stop.

Because Frankfurt (we stayed downtown in Cologne and in Munich) is famous for periods of expensive hotel prices, we stayed at the end of an SB line, but we mostly did car daytrips away from the city. That's one reason we stayed out of the city center. But you may be over-imagining the Oma's Quarkbollen (... a personal favorite ... ) hominess of modern-city-suburbs. A postwar developer's reinforced-concrete luxury two-story house (with the required AirBnb coffee-pod machine and scented soap) is not actually like the wooden homes that Band of Brothers bedded down in.

I would also add the Cologne is good for more than one day of visits, and is an unusual city in that so much of it is walkable from the HBF (... or a nearby, comfortable hotel, like the Hilton we used.) But let's say you want to go to Duesseldorf or Monschau? What a drag to have to go into Cologne first.

A budget is a budget, and I'm not making fun of that. But you need to add up all the actual costs, and the value of your lost time, in making this kind of plan. I note that some budget group tours put people in suburban hotels because they're cheaper, and easier to load the bus in the morning they leave. But it's not the best way to "see" the alleged destination.

I agree that in terms of train connections, you are better off near train stations. But if you don't mind a half-hour regional train into the city centre, then taking a train back out you can find a town like Freising near Munich to be a nice option. Trains headed to other places often stop there, so it might be a nice option, and the town itself is nice. But ultimately our experience has been that when we travel, being central is better.

Sankt Goarshausen is a good base near Cologne. You can get a ferry over the river to Sankt Goar and there are good train connections either side in both directions.