Doreen,
My suggestion would be to travel in one direction by boat, as it provides a beautiful look at the picturesque towns and vineyards from the water, and make the return trip by train. The Belle Epoque steam paddlewheeler La Suisse is an elegant way to make that trip, and you'll be dropped at a dock right next to the Chateau. Depending on where you're staying in Lausanne, you'll need to get to Ouchy using the Metro (Funicular) and the waterfront to connect with the boats. From the Metro station, cross the street and head towards the boats and you'll find the ticket office. If you anticipate being at the Chateau over lunch time, you might want to pack a lunch as there are no food services at the Chateau except for some vending machines. The food services on the boat are limited to a small cafeteria at the front with packaged sandwiches and that sort of thing, and an elegant dining room at the back which is dreadfully expensive.
If you want to carry on to Montreux after your tour of the Chateau, walk about 200 meters to the Bus station at Veytaux, château de Chillon for the short ride to Montreux. When you've finished with your visit there, take the train back to Lausanne (only about a 25 minute ride).