What time of year will you be there? I was there last June and the flowers along the lake were gorgeous. And I don’t even like flowers normally. 😊 5-7 hours sounds good. Depending on the boat schedule and your arrival time, you could walk around the lake and then take the boat to the castle. Or take the boat first and walk around when you get back to Montreux. You can also walk around the lake near the castle as that won't be as crowded.
For me, the castle, boat ride, and walking around (or just sitting near) the lake are the highlights. There’s a train ride up a mountain you could do, but that would be too much for one day.
If you click on my name, there’s a link to some of my photos. The ones 5 rows down are from Montreux—the castle, boat and lake. The boat goes right by the castle, so you get some good views. And of course there’s some great views from land too. I didn’t end up going inside the castle because I was saving that for a rainy day and I lucked out and never had more than a few drizzles.