The Rhein boats are a lot slower than the trains, particularly going up the river (to the southeast), against the currenct, from Koblenz to Bingen. Here is the schedule. For instance, from Bingen it takes about 1h25m to St. Goar or 2h50m to Braubach. The Marksburg, in Braubach, is the only intact castle on the Rhein. All of the others have been destroyed and most have been rebuilt, not as castle. Rheinfels is a ruins but very big and interesting to explore.
Whether you can do it all in a day depends on how long you spend at each site. I did St. Goar and Bacharach in a day along with a 70 min boat ride from St. Goar to Bacharach, but I spent almost three hours at Rheinfels and another hour at Oberwesel. You can only get into the Marksburg for a guided tour. English tours are at 1 and 4 pm. There are tours in other language at other times; they will give you an English script of the tour.
And you don't have to spend all of your transit time on the boat. There are rail lines along both sides of the Rhein. It only takes 21 minutes by train from St. Goarshausen (across the river by ferry from St. Goar) to Braubach.
"Is there any way to get a single ticket from mainz to koblenz and hop on and off as you please?"
There are hourly milk run trains, Regional Bahn or RB, on the left bank of the Rhein from Mainz to Koblenz, plus a Regional Express trains, RE, every two hours, but the Bahn site is programmed to show faster trains. If you click on "Change", then "Change other data", then check the box for "only local transport", the site will show you only the regional trains on that route. If you buy a Rheinland-Pfalz-Ticket for 24€ single (29€ for 2), it will give you unlimited on/off travel between Mainz and Koblenz all day long (on workdays after 9 am).
For Mainz to Köln (Cologne), you can have unlimited travel on any regional trains anywhere in Germany for 44€ for one person on weekdays after 9 with a Quer-durchs-Land-Ticket or 40€ for weekend days anytime with a Schönes-Wochende-Ticket.
There is also a rail line on the right bank, but you have to select "Wiesbaden" as a stopover to see those trains. Same ticket pricing applies.