You will be fine in either dresses or shorts. It can be warm depending on when you go, My experience is in late September. Lots of the hill towns have beautiful churches where you may be more comfortable in a dress. My wife kept a sarong her day bag and wrapped it around her shorts when she felt the need for a dress. Check the weather. It can get chilly at night, especially when the wind blows. Have lunch on the Cassis harbor and then take a boat ride to see the Calanques, cliffs carved out by the Med. They have a short, medium and long version of these, each costing a little more. The medium one was enough for us. You didn't say where you are staying, but when we were there the further you are from the coast it is higher and the cooler it gets. I'm going back solo in September after taking my family there in 2013. Don't miss the Pont du Gard (going to canoe under it), Aix en Provence, Nimes, Arles, Orange and my biggest surprise, Gordes, which was recommended by a friend on the way to the airport. They have a cool market day if you can find out when it is. We rented six nights in a farmhouse in St. Remy just south of Avignon and did day trips like the spokes on a bike every day. Great location.