What is a good small town to stay about 2 hrs out of Frankfurt? One night then heading to Amsterdam. Wouldn't mind something on the river but doesn't have to be.
The Rhine town of Boppard is a very good option.
There are direct regional trains from FRA's Reginal train station (Regionalbahnhof) every two hours. The trip takes about 75 minutes. In between these direct-train journeys there are journeys with a transfer in Mainz.
Königswinter at river Rhine (via Siegburg) - opposite side of Bonn. Drachenburg castle is very nice.
If you're going by train stop in Koln. It's an interesting town, the area around the river is great at night, and the cathedral is always worth a visit. You'll have a lot more options for lodging, and it's where the trains change (depending on what ticket you have.
Just as info: Köln / Cologne is a city with more than a million people - so not a small town as requested (by German measures).
True, Koln isn't small. But it's on the river and where the ICE trains change over. It's the mid-point that makes the most sense if going by train. If you have a rental car it's going to be very expensive to drop it in NL unless that's where it originated.
Since the specific river isn't mentioned, smaller towns that are worth visiting on the way between FRA and AMS would be Maastricht and Liege.
I mentioned Boppard as a solid default-base for a small-town stay - but also because it lies within the scenic Rhine Gorge (Upper Middle Rhine Valley, a UNESCO World Heritage Site) and shares the riverbank and railway with other small, old-world towns like St Goar (approx. 10 min. from Boppard,) Oberwesel (15 min.,) and Bacharach (20 min.) Marksburg Castle, one of the few Rhine castles which can be toured, is found just across the river and north of Boppard in the small town of Braubach, which is reachable in a number of ways (30 minutes on a K-D cruise boat, or take a train via Koblenz, or use the Boppard ferry to Filsen + train to Braubach.)
You could actually stay in any of these small towns and visit the others, but Boppard tends to offer certain advantageous as a base town.
Watch this drone video for views of the entire Rhine Gorge .... Marksburg is at 5:34:
https://www.youtube.com/watch?v=2XElvTqJVWA