Bus 715 runs between Melk and Krems with a bunch of stops along the way. Travel time is just over 1 hour. To find the timetable, go to www.vor.at and enter Melk Bahnhof as the starting point. Select 715 from among the list of bus routes displayed. That will give you the schedule in both directions. Be sure to use the section for the correct day of the week. It appears the schedule is the same year-round.
Although Melk-to-Krems is the recommended direction for boat travel, because that's the way the river flows, it doesn't matter which way you go on the bus.
Durnstein proved a picturesque place to stop along the way; get off the bus at Parkplatz P2 in Durnstein if you want to explore there. That's where we switched from boat to bus. I haven't taken the bus from Melk to Durnstein, so I can't compare bus scenery to boat scenery. However, the walk to the dock in Melk from the train station or the Abbey takes a bit of time, and with the bus you won't need to do that. The bus is much less expensive than the boat.
There may be a day-ticket for the bus, which might save you money if you stop along the way.
For two or more people traveling together, an Einfach-Raus ticket will reduce the cost of the train. It cost us €36 for two people for the round trip. With that ticket you need to take one of the regional trains rather than an express. The regionals to Melk leave Westbahnhof in Vienna at 18 minutes after the hour. I think on weekdays the 09:18 train is the first one you're allowed to take. Trains for Krems depart from (and return to) Franz-Josefs-Bahnhof.
Rick has a very useful write-up about this trip in his guide for Austria. I don't think he mentions the photography prohibition at the Melk Abbey, however.
For me the Wachau Valley scenery is pretty but not particularly dramatic (even from the boat). It's not equivalent to the Alps or the Norwegian fjords. It's a nice opportunity to see one or more small towns during a city-oriented trip, but I think it's a costly outing if your only real interest is in the scenery. The value of the trip goes up considerably if you're really interested in seeing the Abbey in Melk or want to do some wine-tasting