Annecy in late November

I wouldn't call Annecy stunningly beautiful. The town itself is fine and has some decent old architecture, like countless other French cities, and a picturesque reach of river with, if I recall correctly, a canal, but that by no means makes it unique. It's situated next to a lake with modest-sized mountains in the background that is pretty but not particularly spectacular as alpine lakes go.

Personally, I think your time would be better spent in Lyon, but at least -- in November -- you won't face the punishing crowds and traffic in Annecy that persist through much of the warmer weather months. If you're really committed to seeing Annecy, you could rent a car in Lyon and make the 2-hour drive to Annecy, spend a few hours, and head back to Lyon. We took guests from the U.S. to Annecy several Decembers ago for lunch and spent a hour or so in the afternoon before returning home to Lyon, and at least the parking hassles that are common in the summer aren't a problem at that time, and I suspect they wouldn't be a problem in late November either.

I visited Annecy twice this year — both times in the summer (I live in Paris and took the TGV). The town itself is quaint and tourist-y, and the canal is pretty. I really love the lake, but it’s probably going to be too cold to swim in November. It is fed by underground springs, so in the summer the water is quite warm. I didn’t have a car, but I would imagine that a drive through the mountains would be amazing. My husband completed the Etape du Tour there in July (a stage of the Tour de France), and the photos he took showed how beautiful the area is.

. I know Annecy is known for nature and sports

I've made several visits to Annecy, including one for a week, and I wouldn't say I was there for nature or sports. I think it's a spectacularly beautiful city, and while I wouldn't choose November to go, I don't see any reason why you shouldn't. I've always gone in the Spring, and while there was a lot of traffic at times, I didn't have a car, so it hardly mattered. And I would not recommend having a car unless that's the only way you feel you can get there. You simply don't need it for a stay of 2 days. By the way, here are some pictures I took in Annecy in March. I know that March isn't November, but it gives you an idea as to what it is like even in cooler months: http://mainelywinenews.blogspot.com/2012/03/first-stop-annecy.html

Annecy is really a pretty city and there is a trail that goes around the lake for walking or biking. I'm not sure it would be nice in November. It's in the mountains and weather is likely to be variable and cold, which limits any outdoor enjoyment.

You could either stick to cities, which are always open. Lyon and Rouen are convenient by train. They aren't as pretty but will be lively even in November.

Chartres is a nice small city that I found very pretty. It's convenient enough to be day trip by rail from Paris but worth staying a night or two.

To go further south will take more travel time and may not work for you.

We visited Lyon and Annecy during Thanksgiving weekend last year. I absolutely ADORED Annecy! It has the quaint, medieval charm of a French village with the unique canal that straddles the city. The lake and nearby mountains are so lovely. If you were going a few days later, I would 100% recommend the trip, because their Christmas markets begin the last weekend in Nov., I believe. We really enjoyed the markets last year, and it contributed to our enjoyment in the city. However, since you will be there just before this, I would put my recommendation down a notch. Still, I think you should go.
I will say that Lyon seemed to be very interesting. We didn't have time to see the cathedral or sights there, but it was also a worthwhile destination, that I'd love to explore more thoroughly. Lyon will be on the itinerary for our next trip to France!

https://marche-de-noel-annecy.com/ FYI on the Christmas markets, just in case your plans change