I spent May of this year in southern France. I was based in Nice the first 16 days, then I worked my way westward, ultimately reaching Collioure (almost on the Spanish border). I wouldn't hesitate to repeat that timing, but I would say there was a bit less sunshine than I had expected. The temperature was fine, but I had a couple of days of really heavy downpours (one early in the month, one much later), and there were some light showers on a few other days. Locals told me that May weather is somewhat unpredictable, and I don't think I'd recommend that month for a sun-worshipping beach lover. I was a bit puzzled that the receptionist in Nice pushed me so hard to go out and see some of the city on my jetlagged arrival day; looking back, I realized that from the weather standpoint, it was the best day I had in Nice. He knew that weather was to be cherished and taken advantage of.
Toward the end of May it was getting hotter in southern France; if you dislike heat, you may want to avoid the very end of May.
May is nowhere near the tourism peak on the Riviera and in Provence (that would be July and August, for sure), which makes for comparative ease in finding hotel rooms unless you run up against a significant local holiday or event, as I did in Arles. There will be some attractions still on less-than-peak schedules, so they may be closed two days a week and may not be open as late as will be the case in July and August. There was one small museum that wasn't even open for the season yet, and a few excursions may not yet be running in May. This could complicate your efforts to get to small towns (Gordes, Rouissillon, Les Baux, etc.) if you opt not to rent a car, because public transportation to the smaller spots in Provence is rather limited. Les Baux may not be accessible by public bus in May; I'm not sure when those buses begin running.
I have not experienced the mistral (very cold wind from the north), but I believe it's more of a risk in April than in May and more likely to affect the western area than the far southeast coast. This could be a risk if you're focusing on Provence, so perhaps do a bit of Googling to see what advice turns up.
I agree that Normandy is a great place to spend more than a day, but if you have time at least for a full-day D-Day tour, I'd say it's worth the slog up from Paris. I really do recommend one of the van tours (I used Overlord) rather than simply driving yourself around, assuming you are not a WW II historian. The museums in Caen (huge), Bayeux and Falaise are all quite interesting. There are others scattered around.
I was in Normandy in July of this year, spending a total of about two weeks in Normandy and Brittany, and it got up to about 86 F only once. That area was a welcome respite from the heat I had been experience elsewhere in France. Given what I encountered in mid-summer, I'd expect coolish, cloudy days in both April and May, with cold days possible in April if you're unlucky. The weather chart on Caen's Wikipedia page says it rains, on average, one day out of three in both April and May. And coastal wind can be a significant factor in how comfortable you are.
In 2015 I flew into Rome on May 25 and was told that the weather had very recently been unseasonably cold (and I think also wet). I got rained on significantly two times during my first two weeks (Rome/Orvieto), but the temperatures remained comfortable during my stay. I, personally, would not go to Rome in April.
There is 15 years' worth of actual hour-by-hour historical temperature graphs for cities in the areas you plan to visit on wunderground.com. I feel like that gives me a better grasp of the range of possibilities than the averages in weather charts and anecdotal evidence like mine.