Just a thought. Many experience travellers to Greece suggest you get beck to Athens at least the day before your return flight. Many of us get out to our first island immediately on arrival and do Athens for 2 or 3 days at the end of the trip.
Not sure if you have flexibility in your planning but I would plan this trip by flying immediately to Crete when you land in Athens.
You should be able to get flights into the evening to Chania or Heraklion. Chania in particular is a lovely place to get over jet lag. The old town and Venetian harbour is just amazing do try to find a hotel within the old city walls. Crete is further south so will be warmer than Milos.
When you are finished in Crete you can ferry from Heraklion to Santorini and make a connection to Milos.
At least you will be able to see the caldera from the sea.
The tourist season starts later in Milos primarily because its off the beaten path from the Cyclades island direct ferry routes. When we were in Milos it was early June and it was still pretty quiet.
Milos is lovely. The landforms are great and there is lots to do. I suggest you plan for a round the island boat excursion ( there are also half way around the island tours as well. You see most of the major sites but get a longer time at the stops along the way.)
Good choice to base yourselves in the village of Plaka. It is on a mountain with amazing sea views. The sunset from Plaka or even higher up the mountain to the Venetian fortress provide views that rival the Caldera sunsets on Santorini.
Near Plaka you will find the seaside village of Klima. Its a row of coulourful boat houses with apartments above. Its a must see location. On our second visit to Milos we had a car and we found out that there are 5 Klima type villages on Milos. We rented a car and had a day where we identified and then found each of the villages.
There is a ruined Roman city in the valley below Plaka. There are catacombs, the amphitheatre has been restored. Its well worth a visit.
You can do a short island hop to the nearby island of Kimolos. There is a direct ferry regularly through the day.
After Milos enjoy a few days in Athens. By then you will have acclimatized to Greece and you will be ready for all the Antiquities in Athens.
Milos has lots of beaches Sarakoniko is one of the most popular ones.
We stayed at Studio Betty. It was pretty basic several years ago but boy does it have great views. It is perched on the edge of the village and has 2 units. If you can get both units that would be perfect for 5 people.
Here are my images of Milos
Milos https://www.flickr.com/photos/stanbr54/albums/72157634779985764
Milos 2 https://www.flickr.com/photos/stanbr54/albums/72157632109429236
Chania 2022. https://www.flickr.com/photos/stanbr54/albums/72177720299646015
