It's conceivable that there will be a tour from Glasgow that includes Rosslyn Chapel, but the cheapest way would be to take the train to Edinburgh and then Lothian Bus 37. I believe the bus takes about 45 to 60 minutes, then there's a short walk to the chapel (5 minutes or so). As a general rule the websites of tourist attractions in the UK do a superior job of providing access information via both private and public transportation, and so it is in the case of Rosslyn Chapel. Do note that photos are not allowed inside the chapel, and hours may be adjusted on occasion (I assume for things like weddings), so check the "News" section of the website before heading east to avoid disappointment.
A plus for public transportation on this trip is that with an early start you'd have time to see a few things in Edinburgh before taking the train back to Glasgow. On the other hand, there's plenty to see in Glasgow for a lot more time than you'll have. If your interest in Rosslyn is architectural as well as or instead of religious, you'll want to try to see some of the Mackintosh architecture in Glasgow.
With four people, I wouldn't discount the rental-car option if one of you is prepared to deal with left-side driving. However, I have no idea about the easiest place to pick up a rental car and how easy it would be to drive out of Glasgow. I was at Roslin in August and I don't remember a full parking lot near the chapel, so I think that's a problem you won't have.
I liked the chapel and thought the audio guide was useful. The chapel is so highly decorated that it takes more time than you'd expect to take it all in. There's also a resident cat, always a major plus for me.
I'd say don't waste your time trying to find the ruin of the old castle (which one walks to from the chapel, but I didn't see sufficient signage to make the path clear); there's not much there but one wall.