I'm sorry to hear about your mom's illness and understand that you want to see as much as possible, but 3 days is an incredibly short visit and you'll find yourself very rushed - Scotland is bigger than it looks on a map and it can take much longer than many people think to travel between places, particularly in the Highlands. As such, you simply don't have time to do side trips to the islands etc.
If the main aim of your trip is genealogical and you family came from Glasgow then I'd tend to suggest focussing on that area, not bothering with Inverness (which whilst pleasant enough, is fairly small and not packed with things to see) and fly home from Glasgow instead.
Glasgow is a large sprawling city so - even though public transport is good - if you want to see places associated with you ancestors you may want more than 1 full day. Do you know where in Glasgow your family came from?
There are also a number of museums etc in the city that might provide some historical insight into your ancestors lives - i.e. the tenement house, the people's palace, the provands lordship, the Riverside (transport) museum, perhaps summerlee industrial museum in Coatbridge or new lanark world heritage site.
You could still see the Highlands with this approach by taking a day tour from Glasgow with the likes of rabbies or Discover Scotland - this would let you see a lot. You could also take a day or 2 trip on the W Highland train line.
If you're sticking with your original itinerary I'll answer your question about train tickets - basically it's complicated, but the short simple answer is generally no - you'd have to buy separate tickets.
Given you said that your happy to hire a car then I'd strongly recommend this option over public transport tbh, as it will be more flexible over public transport and save having to clamber on and off trains and busses with luggage etc. One you are finished in Glasgow then hire a car from the airport (to avoid driving in the city centre - not recommended imo). To get to Inverness you've got 2 main routes, either:
The A9 via Stirling, Dunkeld, Pitlochry, Blair Atholl, Aviemore etc. (Similar route to the train).
Or the (longer) A82 via loch Lomond, Glencoe, Ft William, Loch Ness etc.
They're both very scenic with nice places to stop, but I'd tend to say the A82 route may have more striking scenery. Take a look at the places online and see where you'd like to see and stay overnight. You could do these routes in a day but it's rushed with a lot of time in the car on busy twisty roads so better done over two days, particularly as the light is getting shorter in October.