To echo the opinions here I would recommend taking a cab from the train station to some place like I Portoni della Brà - the gate at Piazza Bra and it's straight shot from the train station. Just the other side of the gate is Piazza Bra and the Roman Arena. From there wander through the old town and check out the lively scene in Piazza Erbe. Grab a coffee or a spritz and then head back to the station.
There is a taxi stand in Piazza Erbe otherwise be sure you know how to call a cab to get back to the station. There are several competing apps but how well the phone works is based on how good your Italian is. If you get stuck consider popping into a hotel and asking them to call one for you.
Whether it is worth it or not is up to you. I found Verona instantly charming and even a very short visit would have convinced me to go back in the future. On the other hand you need to check your luggage and your leisurely stroll would (at least for me) having a bit of ticking clock on it to be sure you get back to the station in time. And I can't think of a better city with a couple of hours to kill wandering the streets than Venice.
It seems unlikely but if you find yourself in the city I highly recommend Verona Luggage Room on via Stella. Run by a great guy who was eager to offer help in enjoying Verona and called us a cab and told us where to stand to catch it. (5/5 would store luggage again.)
Either way enjoy your trip,
=Tod