"Thus far our itinerary is to arrive Cologne the morning of the 24th...Then see the valley on the 25th..." Marksburg Castle, Rüdesheim and most of the other handsome Rhine Valley towns lie south of Koblenz, a healthy distance from Cologne, and it will take you most of the day on the 25th to do your exploring. Your plans for Cologne sound brief. So maybe you get an early start on the Rhine Valley and leave Cologne in the pm on 11/24...
The Middle Rhine Valley towns generally will be in varying states of hibernation. St. Goar will be deadsville (there are lots of empty storefronts there anyway these days.) Larger and more desirable is Boppard, but even there don't expect much to do in the evening. The one town that should be very peppy is Rüdesheim - it has a very attractive and popular Christmas market that opens 11/24. But you don't necessarily want to stay there on the 24th as R'heim is at the southern end of the valley and you'd be backtracking on 11/25 for your sightseeing - and rooms may be at a premium there. For overnight, I would probably opt for Braubach (home of Marksburg) or Boppard, both very nice places.
Marksburg Castle is stunning and offers the best castle tour in the Middle Rhine Valley no matter the time of year - and in late November, it may be the only place to get inside one for a look around.
Even if it were dark on the drive from Cologne, you wouldn't miss out on much scenery, the best of which is from Braubach south. The last tour of Marksburg castle is at 4 pm, so either you take your tour there Mon. afternoon (which would free up time on Tuesday for sure) or the next morning (opens 11:00.) Braubach is quite small but has a very fetching old town quarter and several places for an evening meal. The Gold Key (zum Goldenen Schlüssel) Inn will be closed but there are other room options in the cozy old town area like Gästehaus Wieghardt and other eateries as well, including Italian and German options.
Braubach accommodations
If you choose Boppard instead for your overnight spot, then you have choices to make...
a.) Drive from Cologne straight to Boppard on Monday for the night, then drive to Braubach Tue am for the castle tour (opens 11:00) OR...
b.) Do Marksburg Castle first on Monday by 4 pm, then drive over to Boppard for the night.
A car ferry operates between Filsen (just south of Braubach) and Boppard between 9 am and 18:00. There are other ferries to the south as well if you wish to cross the river elsewhere (no bridges between Koblenz and Mainz.)
On Tue, I would definitely find a way to fit in a visit to Rüdesheim's Christmas market (R'heim is south of Braubach on the east bank) and to see Bacharach and Oberwesel (just a few minutes apart, south of Boppard on the west bank) before the lengthy drive to Heidelberg.