So, you have 2 responses and 2 somewhat differing opinions.
Bottom line, it comes down to your tolerance for risk and how good you are
at making your way around a place you may not have been to before.
Obviously, if you are delayed getting into Zurich, you should probably bail
on the go-into-Zurich idea.
I would offer that getting into Zurich is not the concern; getting back is.
Depending on the wanderings you take downtown, you may take a
different route to get back to the Bahnhof. That's where getting turned
around/delayed becomes a concern.
Since you have a long flight (or two) ahead of you after the stopover, if
it's hot outside and you get all sweaty and worked up, is that the state
you want to be in for a long flight?
You probably have 2 hours available to you downtown (I'm swagging 1
hour to get out of airport/downtown, 1 hour to make it back from your
destination to the airport, and 1 hour for security/immigration).