Itinerary and transportation advice Western Ireland

I'm not sure you need a car.
According to Google, there is a bus to Ballinasloe, which has a train station on the line to Galway. In the other direction, the next stop is Athlone. It looks like you will have passed through Athlone on your boat tour. If not, there are quite a few things to see there. I haven't been yet, but it's on my list. There are rental car offices there, but you can also get the train to Westport (or Castlebar, which has a car rental office). From any of these towns, you can take the train back to Dublin.
Even if you're not staying with your relatives, maybe you can persuade one of them to give you a driving tour? Maybe there is a young person who would like to earn a little money. Failing that, you could hire a local driver. When my friend and I arrived at the Westport train station, there was a van driver from McGing's picking up some other folks. He very graciously offered us seats, as well (I mean, we paid, but only €10 for two) for a ride to our accommodation on the Quay. https://www.taxiwestport.ie
There are, of course, other operators in Westport.

Also, this is our first time driving in Ireland or anywhere in the UK.

Are you actually going to the UK? If so you will all need ETAs attached to your passports.

We are not going to the UK. I was referring to driving on the opposite side of the road for the first time. Thank you for reaching out.

For music, there’s Ennis. Unfortunately, the fantastic, big, 10-day Fleadh Nua music festival, which we attended 3 years ago starts tomorrow, and ends June 1. So it looks like it’ll be over by the time you’re done with the houseboat. There are still great pubs that have sessions throughout the year.

On an earlier trip, we stayed for a few nights west of Galway, in Clifden. Between there and Cong is the falconry experience at Ashford Castle (did you do that while in Cong?), where you fly your own hawk with a guide, and a big leather glove on your arm. Neat! You might not want to stay so far west as Clifden, but maybe do the hawk walk on your way towards Westport.

So on the way to Westport, there’s Leenaun, located on the only true fjord in Ireland. We took a 2-hour cruise out on the fjord, and had mussels onboard that were pulled out of the waters there. Maybe your houseboat time means you’re not looking for another boating experience, or maybe this would add to it. Maybe spend the night there?

Closer yet to Westport, and actually a bit beyond it, there’s Achill Island. It’s connected to the mainland by a bridge, and would offer a different experience than staying on mainland Ireland. The Aran Islands (which I’d recommend with more time) would be probably too complicated with your timeframe this trip, but staying on Achill Island would be feasible.

As for a car, my guess is that you’d need to backtrack down the River Shannon from Portumna to Limerick or Shannon Airport, or over to Ennis to get one.

Thank you. This is very helpful advice.