I didn't know this existed:
https://www.compagniecorsaire.com/en/destinations/entry-267-dinan-and-the-rance-river.html
At 2:45 one-way (as I read the French), it's longer than driving. But I agree that on a nice day it could be lovely. August could be very hot, even on the coast. It's not clear to me how long you could spend in St. Malo, which is worth a couple of hours itself. This is the height of the European vacation season, but it is too small a product to know (in the U.S.) if it will sell out every day. You certainly don't want to get stuck in St. Malo without your car, if you are sleeping in Dinan.
Do I understand the timetable to say that there is ONE departure from Dinan on August 30, at 9:30AM, and a return from St. Malo at 18:30? The brochure recommends reservations because of tidal and river conditions: " Il est donc recommandé de réserver pour avoir confirmation des dates et horaires."
You are also asking if St. Malo is worth an entire day. In season, with the massive luxury marina to walk around, as well as inside the walls and the beach, it's just worth a day. But if you only have one night in Dinan, you have to get half a day for Dinan. Both towns could be said to be touristy, but I liked St. Malo better. That's very much an opinion.