If you are not interested in archeological/historic sites then Athens may not be for you. However, there's more to Athens than "ruins". I would do a lot more research on the city because while it's not Rome it does have some very interesting areas of the city and is worthy of a visit.
Four days is really not enough time to spend. It takes time for ferrys to get to Santorini and return. A flight would be quickest saving lots of time on a boat.
As stated earlier you have to go through Athens to get to Piraeus for ferrys so you may end up in Athens whether you want to or not.
If you want a real nice town that's only 2 hours or so from Athens then head for Nafplio in the Peloponesse. It's one of Greece's prettiest towns, lovely waterfront, beautiful Old Town, historic sites right in town and while you may not be interested in archeology it's a short distance to two world-class archeological sites, Epidavrous and Mycenae. You can also do a day cruise to Hydra/Spetses and get back to Nafplio for night out.
I think that might be a good option. It will save time on getting to/from Santorini, get a feel for a lovely town, excellent tavernas, pleasant walking and a small beach on the back side of the town.