So when you say, "...the places in Germany/Alsace we might be interested in...," I am thinking you might want to post your request for help in the FRANCE forum as well, where the Alsace and STRASBOURG are.
Strasbourg of course is not a "small, cozy" place. Since the Alsace hugs the German border, you could actually stay nearby in Germany (as you are asking to do) but not in places so far away as Nuremberg, Würzburg and Rothenburg. Germany is blessed with Christmas markets and nice towns all over the place, including destinations near the Alsace.
What if you stayed in the old-world town of Gengenbach (which has its own market) and did day trips from there to the ongoing markets in Strasbourg, Freiburg, and Gengenbach's immediate neighbor, Offenburg? Basel and Colmar would be within day trip range as well, perhaps. Gengenbach is a gateway town into the Black Forest, where lots of cozy villages can be found, many along the scenic Black Forest Railway at higher elevations. Do outings to places like these by train:
Schiltach
Triberg
Haslach
Gutach
Hornberg
The railways in the Black Forest can be traveled free of charge if you stay in Gengenbach, one of the "KONUS" towns:
https://www.youtube.com/watch?v=KkgzMXQ7blQ