We are a family of 4 who gave rented several cars in Europe (estate cars and SUVs). The only size that will fit 5 (plus luggage) will be a minivan (7 seater). Remember that you will likely have 5 carryon size bags PLUS 5 personal items/ backpacks. Plus any shopping you do along the way. It all takes up space.
We are traveling with two family members this summer (6 people total) and have rented a 7 seat minivan. We own a Totora Sienna, and could easily fit our luggage in the back. But the cargo area in European minivans is much smaller. I know it’s going to be a tight squeeze to get the luggage in there, so everyone is limited to a carryon bag, plus backpack they’re willing to hold on their lap, if need be. But we are traveling no more than 3 hours between stops.
Parking a minivan will be a challenge - this is probably why you were told an SUV is too big. Parking spots are smaller in Europe, and be very aware of the height of the vehicle if you try to park in a parking garage. Driving on the narrow roads may be interesting - be ready to pull in your side mirrors.
I absolutely agree that you should not plan to drive immediately after a flight. Take the train to Rouen and pick up the car the next day.