I have done it both ways. One time, with the help of a French national, I was able to take buses to Pont du Gard - and it seemed to take two bus rides each way and about an hour or more each way. So basically its your whole day. Also there is not city near PDG so plan on eating elsewhere.
Driving though was very fast. Maybe 20 minutes. I found a little roadside restaurant and market about a mile away too. In the same day we were able to hit Uzes, Nimes, Arles, Les Baux, and still be back in Avignon before sundown.
The only bad thing was that we did not have a car reserved so it cost about $100 for the day. But if you reserve it in advance it might be $40-50.