I passed by Sénanque abbey 10 days ago and the lavender only had green buds at the tip of the stems, so I would not expect a full bloom for another 10 days or so, that is to say mid-June which is typical for the area.
If you do explore some villages in the Luberon, you will drive by lavender fields. There are a few between Gordes and Roussillon, and there are many along the road between Bonnieux and Saignon (D232) on the Claparèdes plateau. The latter are really stunning, with a great view of Mont Ventoux in the background.
As for Arles: not silly as long as it is not market day! It is not out of the way to Barcelona. With a luggage-full car, use "parking du Centre" car park which is secure indoors and just a short walk to the theater and amphitheater.
On a market day (Wed or Sat), I would not want to deal with Arles. An alternative could be Aigues-Mortes, a beautiful walled city in the western reaches of the Camargue, easily accessed by fast roads (you rejoin the main highway to Barcelona in Montpellier).