Can't say anything plus or minus about Dijon. It's been "on my list" for some time and has good reports, but so far I've only been through it on the train. The journey involves a TGV (High Speed Train), so requires booking in advance, and the earlier the cheaper.
Basel definately, Strasbourg definately. Both have good Christmas markets and lots else to see.
My recommendation would be Freiburg-im-Breisgau. It is a lot closer, but not much quicker to get there, because from Colmar you need to take a bus to Breisach-am-Rhein and then a train. It is a beautiful city, with a market surrounding the cathedral. It has a good Christmas market (nearly as good as Basel :-) ). It also gives you a chance to see a different country and culture. You can then say you went to 3 countries on your trip.
I would also recommend the Route des Vins d'Alsace, which includes 3 perfect little towns: Kayserburg, Riquewihr and Ribeauvillé. Except, you need a car for this. And it may be a bit bleak in Winter, touring vineyards is normally a good weather activity. If you do this look for signs "Cave Co-operative" (= Communal winery), or "Dégustation" (=tasting, i.e. you can taste the wines before you buy). But make sure you designate a driver before you go in.
Don't forget most Christmas markets run from the last weekend in November to midday on Christmas Eve (24th December).