Our last trip we spent four nights and still missed so much. I'd love to go back for a week.
We stayed in Salon but it was just a base. We spent a day doing Nimes and Arles, a day in Aix (that was when they had the torrential rain last Oct so half of that day was literally a wash). Les Baux was really neat and Rousillon was gorgeous. You will have no problem filling 4 nights/5 days. Pont du Gard is cool as well. We did Avignon in 2012, but you could spend a day there (and the pope's Palace is nice, but inside is pretty empty and we should have saved the euro we spent).
There were so many other spots I wanted to see - St Remy, wish we could have explored Salon and all these other charming little places...
Not sure if you know of or use l'Occitane products (skin creams, etc) but one of our days I booked a tour at their factory in Manosque, so that was neat as well...but you do have to book those in advance...it was about 90 min or so. And we got free lip balm! :)