From the port of Lavrio it is possible to go by daily ferry to Kea in the Cyclades (1 hour ride), which is not possible from the port of Piraeus.
Weekly ferries also serve some islands in the North Aegean Sea such as Agios Efstratios and Limnos as well as other islands in the Cyclades.
Due to its proximity to Athens Cape Sounion is more of a day trip destination than a long stay. Not sure many people have stayed there overnight.
Within driving distance there is the Archaeological site of Thorikos (north of Lavrio)
https://www.gtp.gr/TDirectoryDetails.asp?ID=14685&lng=2