First time trip to Ireland - self drive.

This is a good place to start:
https://www.ricksteves.com/europe/ireland/itinerary

There is quite a bit of other useful information in the Ireland section.

That is a very big brief and there are so many options.

With only 6 or 7 days, don't be too ambitious and try and cover too much - Driving in Ireland is slow and it will take a lot longer to cover the distance than you might expect.

Are you coming off a transatlantic flight ? If so allow a day in and around Dublin to recover from jet lag before setting off.

A typical itinerary might be Galway for two nights, visiting Clonmacnoise early Christian site on the way. Explore Connemara (suggestions - Roundstone , Clifden and the Sky Drive, Killary Harbour and a cruise, Kylemore Abbey....) There is also Ashford castle and a falconry experience,

Then head to Doolin for one night - this takes you past Dunguaire Castle and round the coast passing through the Burren which is limestone scenery and very different to the rest of Ireland,] as well as the Cliffs of Moher.

Then head to either Dingle or Killarney for the remaining nights before heading back to Dublin. (Possibles are Slea Head Drive, Ring of Kerry, Kerry Cliffs, Gap of Dunloe (think about a jaunting car ride), Muckross house and gardens, ri=uined Ross CAstle... ...

Depending on the time of flight you may need to book a hotel on or near the airport for the last night. Between Killarney and Dublin, think about breaking journey in Tipperary, Rock of cashel or Kilkenny...

maliaround - You've set up some factors/preferences that work against each other, in particular: you want "coastal and rock landscapes" and scenery. And you have what you're calling 6-7 days. And you are flying in to and out of Dublin, on Ireland's east coast. There's a major tension right there.

Much of the rocky, drop-dead gorgeous coastal scenery that you seek is in the west and southwest of the country. Ireland is small compared to say the USA or Canada, but it's not tiny, and getting across it takes some time (most of a day, even if you're efficient). With "6 or 7" days that's going to be hard, unless you skip almost everything else (it's doable, but at the price of skipping other things). And a lot rides on exactly how you are counting your days. Do you really have 7 full days on the ground (or less)?

I'll ask a painful question: is there any way you could extend your trip to Ireland by (at least) a couple days? If you can, then your wish list becomes at least a little more possible. But if all you have is say, 5 or 6 full days on the ground (not counting your arrival or departure days) then you will be pretty limited in roaming very far. Your inbound flight will probably arrive in the morning (most flights from North America do) leaving you in a jetlagged and sleep-deprived fog for your arrival day. You need to spend your last night in Dublin (or close to it) so you don't miss your flight home.

Do the math and count your days/nights carefully. If you have 8-11 days (full days, not counting your arrival or departure days) then you could range across the island and "do" some of it's most iconic and unforgettable spots on the west and southwest coasts (it would still be a fast-moving pace, but at least it's doable). If you are limited to 5-6 full days, that's going to limit you and probably makes it impossible to get to the far southwest of the country (IMHO the best part, especially if you want those iconic seaside scenes).

Get out a calendar, a pencil and figure out how many days you really can make for this trip. Ireland is worth the time it takes to get out the the southwest (and savor it for a few days). If there is absolutely no way you can add days to your trip, then you can't, but you're going to be within grasp of some really incredible places that would fall just outside your operational radius - if there's any way to add a few days, it's very much worth doing so...so take a look at circumstances and see if there's any way to make that happen. Just my two cents.

As has been pointed out, your real issue is how to make the most of the very limited amount of time you have.
One obvious suggestion is to start your tour of Ireland at the beginning ... which is Dublin. Basing yourself there and doing day trips, either by public transportation or via organized tours, would minimize your running around time - would save you some money too. Possible day trips would be north to Newgrange and the Boyne Valley, south to Kilkenny, or west to Galway - all of which can be reached fairly easily via public transportation. You could also head up to Belfast, either as a day trip or (better) for a couple of nights in order to explore the beautiful city and its Titanic Museum, as well as booking a tour up to the Giants Causeway which would answer your desire to see some gorgeous rocky coastline as well as the evocative Dunluce castle.
That said, the Southwest is famous for good reason - the Ring of Kerry and the Dingle peninsula are gorgeous and in October you'll not have to deal with the tourist crush of summer. Ryan air runs a couple of direct flights daily between Dublin and the Kerry airport near Killarney. Since you'll aleady be at Dublin airport anyway you might be able to continue your journey on arrival day and either collect your rental car at the Kerry airport or (better I think) to taxi into Killarney for a night or two in order to sleep off your jetlag before getting behind the wheel. Staying somewhere along the corridor between Glenbeigh and Castlemaine would satisfy your desire to see some charming Irish towns while being central for short drives up to Dingle or over to Killarney. You could then return your rental at Kerry and catch one of their return flights to Dublin and thence home.

Any chance of flying into Shannon rather than Dublin? It's much more convenient of you are wanting to visit the west...