If you want to purchase each leg separately at full fare, you can do so, or you can purchase a full fare ticket form Salzburg to Vienna. A full fare ticket is an open ticket; you can make breaks in the journey as desired. If you purchase a reservation from Salzburg to Vienna and get off in Melk, you'll have to purchase a new reservation from Melk to Vienna or do the last leg without a reservation.
As for discount, advance purchase tickets, I know you can do this in Germany, and I assume you can also do it in Austria. That is, you can book the ticket with a set stopover in Melk. But both train legs will be date and time specific, so you have to be back to the station for the second booked train. I know how to book a trip like this on the Bahn website; does anyone know how to do it on the ÖBB website?
Lastly, if there are two to five of you, there is the Einfach-Raus-Ticket, for 36€, which is an all-day, hop on/off pass for regional trains all over Austria. With it you can stay in Melk as long as you want (that day) and board any regional train onward to Vienna, but taking regional trains from Salzburg to Vienna is the slow way to go.