I’m piecing together what I think is your itinerary based on what you’re saying. If you hate crowds, steer clear of Zermatt. It’s an absolute zoo of selfie sticks and e-taxis zooming all over the “car free” village. It’s way worse than somewhere like Mürren, which is certainly touristy but not crowd overload everywhere you look. The BO has crowds but they’re also spread out over a pretty wide region, whereas everyone from the BO also goes to Zermatt and crams into one little village. You get the idea.
If the area around Zermatt makes sense with your overall itinerary, go to Bettmeralp or Saas Fee instead. Both car free villages and night & day in terms of crowds. Great hiking and small village life. From Bettmeralp you can hike along the Aletsch Glacier. From Saas Fee you can hike along the views of all the 14ers in a way that certainly tops Zermatt views in my opinion (the Matterhorn has been branded to death, in the end it’s just another mountain, of which there are plenty others to enjoy in its place)
I’d never suggest Zurich, because of the info stated in my original response. If you want to see St Gallen, why not just stay there? You could do a day trip on the lake to see Meersburg & the island of Mainau if you enjoy flowers, nature & charming villages and it’s closer to the Zurich airport than Luzern is.
Of course my opinions are just opinions and may not be for everyone, but if nature & charm is your priority vs checking off main list destinations at the expense of moving with the crowd, I think you’ll agree with my opinions once you experience it for yourself :)