Every country in Europe has different rules on holidays- Austria is one of the more extreme.
On Sundays and holidays (including Whit Monday), all shops, groceries and bakeries will be closed. The exceptions, however, are at the train stations, where they are permitted to stay open. Groceries and bakeries at Westbahnhof, Meidling, Franz Josef, Praterstern and Landstrasse will all be open. A warning, however, these places will be complete mob scenes and should be avoided.
Public transport will be running, but an a slightly reduced schedule (I doubt you will even notice a difference).
Restaurants, museums and attractions will be open as normal.