I would try to keep the ground travel fairly limited with your short time. FRA airport in and out, as travel4fun did it, probably makes a lot of sense. Here's a map of the various wine regions in the vicinity of FRA, Mainz, Wiesbaden and beyond:
https://www.bkwine.com/wp-content/uploads/2021/04/csm_Karte_der_deutschen_Weinanbaugebiete_2015_ee577c518a.jpg
"We're very much open to doing smaller towns and villages if that works better."
Germany's small towns can be very charming to visit in winter, but for booking rooms, be sure to pick a place with a daily Christmas market; other small towns and villages in the Rheingau, Rheinhessen, Mittelrhein, Nahe, and Mosel regions are worth a visit during winter's short daylight hours, but STAYING in some of them can mean a lot of thumb-twiddling once the sun goes down!
On the Rhine: Rüdesheim could be a good small-town base... a popular daily Christmas market, winter cable-car rides, Rheinweinwelt (wine-tasting facility) and proximity to Wiesbaden, Marksburg Castle in Braubach (see 3 pages of pics,) Mainz, Koblenz, FRA airport and Frankfurt, all of which are good for outings that offer general sightseeing and Christmas markets.
BERNKASTEL is probably the Mosel's most stunning old-world wine town. A daily Christmas market as well as proximity to historic TRIER and COCHEM for day trips - and winter cruise options as well - make it a good bet.
https://en.bernkastel.de/christmas-market
https://www.moselrundfahrten.de/en/trips/boat-tours/round-trip/17
As KGC points out, access to wine tasting experiences is a bit slim at that time; you can of course find wine at most dining establishments, but a shift in focus from wine to other things is probably wise. Trier, Germany's oldest city, has numerous options for sightseeing.
https://www.trier-info.de/en/places-of-interest
https://www.trier-info.de/en/museums
Staying in Trier - and doing outings to Cochem and to Bernkastel from there - is also a reasonable option. Luxembourg airport is quite close to Trier, btw.