It is doable, having some stamina. I would do it on my own (and not with a tour group) because of greater flexibility in the schedule or when making last minute changes, e.g. due to the weather.
To save time I would take the fastest trains available between Munich and Salzburg, i.e. the RailJets (operated by OeBB). The ride is 1:30 hours each way, so having gained one hour in total.
The arrival time of 11:30am in Salzburg is far too late for all what you have planned. Best would be to arrive around 9am at the latest (e.g. RailJet departing from Munich at 7:24am).
Visiting the old city and Hellbrunn may cost you two hours easily. Then you must catch the bus #150 to St. Gilgen, catch the ferry to St. Wolfgang, and possibly wait for the next train up the Schafberg (by the way, Austria's steepest cog railway). At the end you must go all the way back.
Yet, we have not counted time for having lunch, for visiting St. Gilgen and St. Wolfgang, or spending time at the Schafberg summit.
You should have all the timetables printed out and with you, i.e. RailJets, bus #150, Wolfgangsee ferry, Schafberg railway. Try to find a itinerary for you without wasting too much time just waiting for the next public transport.
Stretch your visit in a way to arrive back in Salzburg around sunset. Maybe you can have dinner before going back to Munich. The last RailJet to Munich departs at 9pm, the last other express train at 11pm.