You'd probably only need a few hours tops at Pont du Gard - unless you want to do some of the hiking in the area (or canoeing on the river) - we've been twice - first time was with a tour that went to Uzes/Arles/PdG - and we only had about 45 min at the Pont - so it was very quick. We returned last year and spent - actually - looking at the time stamps on my photos - an hour. We got there after the info center was closed, so if we had wanted to see that maybe add another 20 min. We walked all around, stopping to take photos and walked a steep path thru the woods to see it from the top.
I really didn't get much of an impression of Uzes in 2012 - it was part of the guided day trip and we only had 45 min to walk around. We didn't get back last year. I remember we arrived - must have been about the time everyone was heading home - and the traffic was backed up, and the parking was almost non-existent - the driver parked to let us off (in a no parking spot - he was going to try and find somewhere to park) and got told to move pretty quick by some business owners.
I second Rousillon - such a gorgeous place - even to walk around for an hour. The colours are beautiful. But I love taking photos, so maybe I'm biased towards picturesque.
We enjoyed Les Baux (if you like historical stuff) and spent about 45 min at the ruins at Glanum (on the way to Les Baux).
We got lost in St Remy (via a GPS that kept trying to send us down a one way street the wrong way - I think there was construction going on and route changes) and after spending a frustrating amount of time trying to find our way out (we were heading for Les Baux) and driving around in circles - hubby vowed not to return ;)
When we went last year, we did Arles and Nimes in one day - the arena at Arles and wandering around the city for an hour, then to Nimes for mid-afternoon meal, the arena, Maison carree and a walk thru the park. So unless you really want to get into Arles, you could spend a half day somewhere else...
My only comment on car rental - next time we rent, we will avoid indoor parking garages. You may have more luck with a Fiat tho - we had a Citroen Cactus - a little larger vehicle - and we ended up getting a huge scratch down the side of the car (on an otherwise new, non-damaged car with only 10,000 km on it) trying to get around a very tight corner in the garage. Ugh.