Oban or Mallig?

Mallaig
Glasgow
Edinburgh

Are all touristy places. If you are seriously averse to them have you considered other less visited places?

Of the two it is Oban with those specifications.

I can't think of anywhere really for art around Oban, but the town is justifiably known for it's seafood scene- both down by the CalMac terminal and on the North Pier.
But of course Glenfinnan has the Jacobite memorial (it really isn't all Harry Potter) and all along the Fort William to Mallaig line there is a lot of half hidden WW2 history, at quite a few of the wayside stations.
If you go over to Kerrera from Oban don't just go on the southern walk (remember a King of Scotland died at Gylen Castle), go to the Marina at the north end for the Hutcheson monument- David Hutcheson, one of the founders of the companies who are now CalMac (the Caledonian Steamship Company, and David MacBraynes).
Apart from the couple of Castles Oban has a strong WW2 history- visit the War and Peace Museum, and the Youth Hostel to learn more (the Odin room at the Youth Hostel), also Ganavan Sands for the Coastal Command history, including a brand new mural on the side of the toilet block.
Even McCaig's Tower which may seem like a gimmick was built partly to make work for locals in winter. Ironically it was to have had an art gallery.
And do visit Oban Episcopal Cathedral- you will probably be surprised. Right in the centre of town, but few tourists get there.
If you visit the Isle of Lismore, do go and visit St Moluag's Church- until the Reformation it was a Cathedral (originally a Monastery). Also we are always talking about Callums Road on Raasay here, but there is a road with a very similar story on Lismore, missing from all the guidebooks. Also visit the Heritage Centre and there are island tours you can do- https://www.explorelismore.co.uk/
If you don't want tourists get the bus down to the Slate Islands from Oban (Ellenabeich for ferry to Easedale, North Cuan for ferry to the Isle of Luing- https://www.isleofluing.org/about-isle-luing

Both isles have real history there (especially Luing, with it's own sub breed of Highland Cattle, even).

If you want to lose the tourists but go to Iona forget the highly priced West Coast Tours buses, take the early ferry over then the service bus to Iona, and you will be there a good hour before all the day trippers. Do the Abbey first before they all arrive then find other religious places on Iona like the Orthodox and the Roman Catholic Oratory, then go and find the glorious beaches which the daytrippers don't have time to get to. You then have an extra hour or two after they have gone before you need to go back for your ferry and bus. The information centre will be over the moon to give you information about what else historically is on the island.

Honestly you can spend almost a week in Oban and escape the tourists every day.

Take the bus or the train to Bonawe for the historic iron furnaces- an important industrial site.

I'm wondering what the "West Highland Express" is, or what Lynda thinks it is. The trains in the West Higlands are not really expresses in the normal sense, by UK standards they are quiet slow and rattly- and one of the longest rides without a/c

Mallaig itself is tiny, most people travel there for either the scenery on the way or the ferries.

By the way Mallaig as a port barely existed until the very last years of the 19th century, when the herring fishery arrived (quite a common story across Scotland).
The West Highland Railway Extension (Fort William to Mallaig) was built mainly for the 24 hour a day fish trains to all over the UK, with passengers being very much a side thought- for what little passenger traffic there was.
The Mallaig Heritage Centre tells that story, but is unlikely to detain you for long. And the food scene has improved with the opening this Spring of the new deli on Mallaig station.
And, of course, 99% of tourists don't investigate why Fort William is called that- because of the one time fort- of which not a lot now remains. But it was a fort before it was a town.