January first week - Rhine Valley or Rothenburg ob der tauber - 2 nights

Weather cannot be predicted at this point for two random dates in January. But you can count in it being relatively cold.

You are asking about a location near Frankfurt, but that wouldn't include Rothenburg. When you say "We will be in Frankfurt", does that mean you have a scheduled stay in Frankfurt in addition to the two nights you are asking about? Or that in total, you have a 2-night stopover altogether - maybe because of a flight via FRA airport?

One place that's very near FRA airport AND Frankfurt is Mainz, a university town on the Rhine just 25 minutes from FRA by train. It would make a good springboard for a journey through the Middle Rhine Valley and is a good destination in itself. Mainz has a small but pleasant old town and is easy to get around on foot with its expanded pedestrians-only zone. If it's too cold to be outdoors for long, Mainz has plenty of museums, churches and other indoor options.

Mainz' old town zone
Mainz' market square
Things to do in Mainz

From Mainz you can reach Bingen in 20-30 minutes by train. Bingen is the gateway to the castles and small towns of the Middle Rhine Valley. You might want to ride the train one day north to Koblenz along the west bank, then return to Mainz southbound along the east bank - and stop in a town or two along the way. Bacharach and Rüdesheim would probably be good choices. Or you might take the train to Braubach to visit Marksburg Castle, which is open year-round.

Yes we have 2 additional nights in the area (we can stay anywhere) we have our outbound flight from Frankfurt. Would you recommend driving? I wanted to stay somewhere in Rhine valley for at least one night explore 1-2 castles and drive up to Cologne and then comeback to Frankfurt.

On average, temperatures are a little warmer in the Rhine Valley, but that doesn't mean at any given time you can guarantee it will be warmer than Franconia (Rothenburg).

Although I think that Rothenburg receives far more attention than it deserves, the math changes in January. Germany is loaded with well preserved old towns, some of which even still have their defensive walls, and several located within an hour's drive of Frankfurt. But in January, many of them can be rather dead. In this case, Rothenburg's status as a tourist draw probably keeps more of the hospitality-type businesses open.

The Mittelrhein can also be a dead during the winter. Some of the castles close, and many of the more seasonal businesses also go into temporary hibernation. Especially if temperatures are a little bit above freezing, the valley can get very foggy.

So... even though it's much further away, I think Rothenburg might be a better choice for January.

But if you choose the Rhine instead, I would consider basing in Rüdesheim. Yes, this is a very touristy town, but it stays a little more alive during the winter than many of the other scenic towns further downriver (Bacharach, St. Goar, Oberwesl, Boppard, etc.). The reason? Many of the restaurants offer live Schlager music, particularly on the weekends. These draw many of the stag and bachelorette parties from Frankfurt, Wiesbaden and Mainz.

"I wanted to stay somewhere in Rhine valley for at least one night explore 1-2 castles"

For a place to stay, I recommended Mainz instead of one of the small Rhine Valley towns - for the reason Tom just provided - there is very little going on in these towns in winter to make it worth staying there. With Mainz as a base town to return to after you've been out "exploring," you'll have a place to hang out with a wide range of entertainment, sightseeing, places to eat, etc.

But the fact that the Rhine Valley towns are dead does not mean you can't visit/explore them. The old town walls and towers of Oberwesel and the attractive half-timbered buildings in Bacharach and Braubach are there 24/7. And it's easy to do from Mainz by train if you want.

"...and drive up to Cologne and then comeback to Frankfurt."

A car will be needed if you want to drive, of course. No reason you can't. A car will make it easier to visit Marksburg Castle too. I don't necessarily recommend driving much beyond Koblenz or maybe Linz am Rhein if it's scenery, castles and cute towns you are looking for. Cologne isn't that at all.

The only Rhine castle open daily in winter for exploring is MARKSBURG in Braubach. See link in my post above. It is an outstanding example of a medieval castle and was never destroyed - it is very much worth visiting. (There is also a palace - Stolzenfels, near Koblenz, which is open weekends only.)

If you do choose to stay in a small Rhine town instead, Tom's suggested Rüdesheim is your best bet. It's touristy and overpriced a bit but it will have a bit more going on than the other towns.

Since you will be in the area the first week of January, you could visit the Speyer Christmas market, which stays open 11:00 to 21:00 every day through the 8th. A Mainz base is best for this trip.