For a non-touristy town, the village of Kilmartin would probably work, but there's also the slightly bigger town of Lochgilphead nearby. Or Tarbert further down the peninsula. Inveraray is wonderful -- I highly recommend Brambles for lodging and daytime eats, and the George Hotel for dinner/pub -- but if you really want to be off the tourist circuit then the former towns would serve you better.
Kilchurn Castle is photogenic from a distance across the loch, but up close there's not much to see. Right nearby, however, is the Ben Cruachan "hollow mountain" hydroelectric power station visitor centre. It may sound boring, but it's actually quite fascinating -- and you can take a tour under the mountain if you have the time.
On your drive back toward Glasgow, along the shore of Loch Lomond you might want to stop for lunch in the village of Luss (watch carefully; if you blink you'll miss the sign) or else at the Duck Bay hotel/restaurant, which has huge picture windows with a commanding view of the loch. When you get to the big roundabout at Balloch, that's your last chance to see Loch Lomond. From there you'll continue south on the A82 (if you need to fill your gas tank, Dumbarton is a convenient place) to the Erskine Bridge and the M8, which has prominent signage for the airport. Note that the airport is actually in Paisley, not Glasgow.
There are a couple of hotels right at the Glasgow airport literally a stone's throw from the terminal. I recommend the Holiday Inn, though the Holiday Inn Express is also passable.
