To do this, you will need to clear Schengen entry passport control in Amsterdam (which you would have had to do anyway for your connection to Vienna, even if you had stayed in the airport, so it is not an extra step) when you land and head to the train. When you take off again for Vienna, there is no passport control, because the Netherlands and Austria are both in the Schengen area; it's effectively like flying domestically within the US or Canada.
It looks like there is baggage storage at Schiphol located "landside," i.e. after you clear passport control and exit into the public area of the airport.
There may also be lockers elsewhere, but in your itinerary you should not use any lockers in the gate area and you should clear passport control with all your carry-on bags. The issue is that if you are arriving into the non-Schengen area, but you are departing from the Schengen area.
These two areas are separated because non-Schengen flights require clearing passport control (in both directions), while Schengen flights do not. If you leave your bag in an airside non-Schengen locker, you will run into problems getting that bag when you depart because you won't be able to return to the non-Schengen area without a non-Schengen air ticket.