Austrian trip...train/bus or car ?

What time of year? Outside your Vienna stay I would take a car. Secondary roads only. Austria is not that big. You can make your own Itinerary. Just be conscious of parking. Innsbruck I would stay in Fulpimes and take the Stubital train into Innsbruck. Take a look at Schladming nahe, Krimml Falls. Zillertal, Lechtal, Silbertal, Virgental, Deferential, Großglockner Hochalpenstraße, Lienz etc. into the Pustertal in Italy. Hard to go wrong.

For me, this would depend on what comes after Austria. If your are planning to go west towards Germany. Then I would start in Vienna. On to Salzburg. Either train or rent a car there to get to Innsbruck. If going north or east from Austria, then I would start in Innsbruck end in Vienna for train connections to Prague or Budapest.

To determine if you need a car, plan your activites around Salzburg and Innsbruck. Do if need a car to do them? Regarding Bratislava rental, check the drop-off fee if returning in Austria. Otherwise you would be driving across the width of Austria twice.

best/more enjoyable

From east to west, border to border, it would take 9 hours to drive across Austria.

People like TTM thrive on driving in Europe.

People like me will do anything they can think of to avoid driving.
So best/most enjoyable is subjective, but i choose the train option.

Check the rental cost in Sopron too.

I love driving thru Austria in my convertible. For me the train is tunnel vision. Each to their own. I guess it comes down to what kind of trip you want. I've been to places routinely by car you will never see by train. Yes, a little more effort and adventurism. But thats a plus in my book, The lesser traveled way.

It's not really clear how far off the "beaten path" you plan to stray. The train and the Postbus will take you almost anywhere, and since I'm a train guy I would always choose them.