Worth renting a car in Barcelona to Montserrat?

You can and it is also well serviced via train. Personally I would do the train.

Don't rent a car, take the train, then the cable car.

Thanks for the advice!

We did this day trip on our own using public transportation and found it very easy.

The day before, we bought a “TransMontserrat” from the TI in Barcelona. Be sure to specify if you want the cable car (Aeri) like Rick takes in his video or the rack train (Cremallera) up to the monastery. I had originally bought a ticket for the rack train, but once I saw the cable car I wanted to change my plans. Luckily, I was able to switch my ticket at the Aeri ticket booth for a small price increase. So glad I did...the Aeri ride is beautiful!

https://bcnshop.barcelonaturisme.com/shopv3/en/product/616/trans-montserrat.html

Some people ascend to the monastery on the cable car and descend on the rack train (or vice-versa), thus experiencing the best of both worlds.

Unfortunately, there isn't a combo ticket to do so and when you purchase TransMontserrat you have to decide whether to use one or the other, thus both trips (up and down) need to be made with the same transportation. But no worries, purchasing a single ticket onsite to, say, descend on the rack train, won't break your bank (~7€).

If anything, note the cable car departs from the train station "Montserrat Aeri" while the rack train departs from the next station, "Monistrol de Montserrat". This doesn't affect your TransMontserrat ticket since you can catch the train back to Barcelona from both with the same combo ticket.

With the combo ticket, once in Montserrat you can also take two funiculars, which are sort of "elevators" (mind me!) to different parts of the mountain: Funicular de Sant Joan and funicular de la Santa Cova:

• The "funicular de la Santa Cova" descends from Montserrat Monastery to the beginning of a trail leading to the revered Santa Cova cave, where the Virgin Mary's apparition is believed to have occurred. It's a favored choice for hikers who trek down to the cave but wish to avoid the steep return climb to the Monastery.
• The "funicular de Sant Joan" ascends the mountain slope from Montserrat Monastery, providing a breathtaking aerial perspective of the entire area. Whether you seek panoramic vistas of the Monastery and its surroundings or wish to commence various hikes within Montserrat Natural Park, this funicular ride serves as an ideal starting point.

Presently, reading R.S. guide book about this day-trip and trying to plan - so thank you for the question! I plan on taking the train.

I have a fear of heights (like climbing the stairs at the Statute of Liberty or Eiffel Tower). The cable car ride sounds beautiful. Is it worth a try?

If you have a fear of highs I would choose the rack train instead. The cable car can be "challenging": https://www.flickr.com/photos/ell-r-brown/48045406932

Take the train and then rack rail to the monastery. I get claustrophobia in crowded areas and my traveling companion is afraid of heights so we did the rack rail train to the summit. I will say that if you want good pictures, go with the train, big open windows and no one blocking your view. Plus it's just as spectacular climb and descent. We were going to take the cable car down but the line was horrendous so we used the train and got to the Barcelona trains before the crowds left for the day.

Take the train.
I found that I needed to have a credit card with a PIN in order to buy tickets.
The train is not Renfe, but FGC Catalunya.