We enjoyed Ile Sur la Sorgue, too. The market is wonderful, but Thursday is a better day to go than Sunday. On Sundays, the market is extremely crowded. Thursday is much less crowded.
I also recommend staying in St. Remy. There is interesting sightseeing: the asylum where Van Gogh stayed for a year and did a lot of paintings. The Roman ruins of Glanum, which I believe is where St. Remy began. St. Remy is very pretty with lots of restaurants and shops, so I think it would be a wonderful place to base yourself.
We also enjoyed staying in Aix en Provence. Our daughter lived there for a few years, so we would visit her and then take day trips to various towns and villages including the Luberon. I think at the most our day trips were no more than an hour away, some were closer, depending on where we went. From Aix, we drove to Marseille and Cassis, to Loumarin, Les Beaux, Gordes, Rousillon. Aix is very pretty with lots of fountains, some old churches, a few museums, and lots of restaurants and shops. You can visit Cezanne's studio in Aix.
Hope this helps.