I agree that combining Normandy and Provence/Riviera in such a short time may be too much ground to cover enjoyably. Google maps is a great resource for figuring driving times. Instead of heading south you could do a loop extending out to Brittany and visit the Loire on your way back to Paris. If you do really want to incorporate the southeast part of the country, the train might be a better way to go than all that driving. (But you do need a car for Normandy).
We did Normandy in two days with a car, but it was too rushed. Once you are all the way out at Mont St. Michel, you are pretty far from Paris and it will take half a day just to get back. It seemed easier driving in France this year than when we went in 2003. Maybe because people have slowed down due to cameras everywhere to catch speeders. Coming into a town around 5pm is not advised, even in small towns the traffic can be crazy. Among the regions of France, Normandy is one of the easier to drive in.
Since you will be heading to the west of Paris, consider both Giverny and Chartres, two highly rewarding stops. I recommend an overnight in Chartres, a city that tries harder to please the tourists and yet manages to retain its own character.
Especially since you have some apprehension about driving, I strongly recommend picking up the car outside of Paris. Picking up at the airport will cost you about $75 extra, but you could pick it up in Rouen for example. Dropping at the airport sounds easy, but we had a hard time finding Orly to drop our car and ended up driving into Paris by accident. Not recommended. If you plan on dropping anywhere near Paris, get a really good map -- we found that the signage is not adequate to find even the airport without one.