Stops in Germany Between Prague and Amsterdam

The trains from Prague to Amsterdam stops in Dresden which people like, I’ve never been. From Dresden to Amsterdam, you can stop in Cologne.

If you've not been to Berlin, it is worth at least a 2 day visit, because of all the history; you can also visit Dresden for a day on the way to Berlin, then continue to Amsterdam.

"I'm especially interested in small historic towns..."

Many good options meet your criteria. I'd be making this trip by train, and there are various routing options, so I'll mention some places that are on or near the route that I'd prefer for checking out small towns. Of course, these places are also accessible by car.

Route: Prague > Cheb > Nuremberg > Frankfurt > Cologne > Amsterdam

Historic towns that may interest you:

1) Places that are easy to visit from a Nuremberg base; some tiny, some a bit larger:

Bamberg
Iphofen
Bad Windsheim

2) Between Nuremberg and Frankfurt:

Würzburg
Miltenberg (involves a small detour)

3) Between Frankfurt and Cologne; use alternative Rhine Valley Railway, the slow/scenic route:

Rüdesheim
Bacharach
Braubach (home of Marksburg, an intact medieval castle)

Thank you all!
These are great suggestions.

Although Google Maps is not a "good" way to plan actual travel, I was interested to see that it offered two "picture of rail car" routes. The southerly one had big Flixbus legs at the start and end, and the northerly one went close to Erfurt and maybe even Quedlinburg.

I don't mean to quibble, but I'd be careful with the term "small historic towns". There are no wooden villages with Private Ryan just turning the corner. We spent a nice day climbing a steep "park" and walking vineyards in Traben/Trarbach, and maybe a stop the same day in Disney-like Bernkastel-Kues. But neither town is worth an overnight, or even a "stop" on a long-distance rail trip. Erfurt is just big enough to be worth an overnight, and Cologne is worth three nights. But it's not a small historic town, it's a superb (war-destroyed) city with Roman roots still visible, and a dozen superb museums, each worth a detour. Berlin is worth six or seven nights, especially with its suburbs.

There is a good chance that KD Rhine dayboats will start early in April, so you may be able to make good use of a stay in Bacharach or nearby. Don't rely on scheules in old posts here, of course:
https://community.ricksteves.com/travel-forum/germany/advice-for-rhine-river-cruise-train

I don't know if it's worth the reading time for you, but this travel report may spur some interest:
https://community.ricksteves.com/travel-forum/trip-reports/baltic-germany-fachwerke-landesgartenschau

You are allowing yourself 4 nights to go from Prague to Amsterdam. Interested in small historic towns...well, how far are you willing to deviate from the route? Is the key word, "historic?"

I would suggest any of the following: Minden an der Weser, Jena, Meißen, Weimar, Soest/Westfalen, Kleve.

I would suggest that in choosing these "small historic" towns (lots of them can be named, all depends on the context) you focus on those towns totally off the foreign/international tourist radar. Those I suggest above fit that , either for the most part or exclusively.

Another suggestion on seeing a historical "small town" going towards Amsterdam, ie if your train goes to Münster (Münster-Amsterdam connection), is the town of Höxter. I saw this place some years back as a day trip from Soest...revealing. The chateau with its immense library is a sight to marvel.

If you're into German cultural history , the author of the poem that was adopted as the German (not Prussian) national anthem, still is , minus one original stanza has his grave site on the premise. Postcards showing him and the original lyrics can be bought in the town.