It sounds didactic, but the Netherlands is currently the name of a country, one which does not include Belgium. I mainly write that because so many posters on travel sites consider the two countries to be the same one, and it's not true, politically or economically. I've spent more time in Belgium than the Netherlands, and would say (an opinion, of course) that Belgium is more interesting!
I like Cologne very much. And I've never been to the Chocolate Museum in two visits. Cologne is good for four days if you like museums and botanical gardens (not gardens in March, naturally). I think it is harmful to a Cologne visit to wake up in another place, despite relatively good public transportation in Germany. You didn't give your home city in your Rick Steves identity profile, but it's like staying in Glen Cove, Long Island to visit New York City. Why do it?
Cologne is too far north for the more picturesque (and mostly stone ... ) towns of the scenic Middle Rhine. Like other posters, I wonder if you mean just "old", or "half-timbered", or "medieval", or just "low-density." Like Belgium and the Netherlands, postwar development has produced vast tracts of single-family, modern masonry homes, replacing the wooden villages that you saw in Band of Brothers. To the extent that such places still exist, they are rare enough to be tourist sights. Unlike the US, there are still plenty of farms, but they are of necessity, modern farms.
Just to ascertain what you are thinking of, I suggest you look at some photos of Monschau. In fact, Germany has similar villages all over, but they are tourist attractions. That's why there is a Fachwerkroute!
I will say that when we attended a flower show in Koblenz some years ago (not particularly near Cologne), we didn't want to stay in Koblenz, so we slept in a family-owned small hotel in Andernach. That has a small medieval town center, and their own history (and what looked like big old resort boardinghouses (??) on the other side of the Rhine.) But I never had the illusion that I was in a history book about Germany. Even when, later in that trip, we slept in Mulheim and visited Trier, Bernkastel-Kues, Traben-Trarbach, Burg Eltz, and later in Belgium, Tongeren, Bokrijk, and Diest, we never had any illusion that we had stepped through a time warp.
I endorse your decision to stay in Antwerp and daytrip to Brussels. I personally wouldn't bother to sleep in Bruges, but I am in the minority here. A parallel idea to the Cologne question (... which I don't recommend ... ) would be to sleep in Turnhout to visit Antwerp. I like Turnhout very much, but there is no reason for a first-time visitor to sleep there, or even make time for it.