I agree that it's very odd for the bus to be nearly full so early. Perhaps that's an error.
Please clarify your schedule. Are you trying to wake up in Lugano the day of your Bernina Express trip? If so, what time does your train leave?
I don't think there's any public-transit option that will be as fast as a direct bus, but it will be possible to get to Tirano. The issue is the timing. You might need to go a day earlier and spend the night in Tirano.
I think one possibility (though I know nothing about frequency) would be a bus from Lugano across the border to Menaggio on Lake Como. From Menaggio there are frequent ferries to Varenna on the other side of the lake. There's rail service from Varenna-Esino station to Tirano. The Varenna-Esino station is uphill from the boat dock, and I don't know about the availability of taxis; I only took that walk in the downhill direction, so I wasn't looking for a taxi. In mileage terms this is by far the shortest route to Tirano. How fast the trip is would depend on the schedules of the various transportation components.
A longer option would be to take a train from Lugano to Como, a bus to Bellagio and a ferry to Varenna. You'd then pick up the train to Tirano from the Varenna-Esino station as described above.
For fewer transfers you'd take a train from Lugano to Monza (just north of Milan) and switch to the Tirano train there.
At this time Google Maps is indicating the last part of the train trip to Tirano will be via a rail-replacement bus between Sondrio and Tirano. That would affect all the routes I've suggested. I don't know whether that situation will still prevail in September.
I think you can make one of those options work, if necessary. Worst case, you should be able to spend a good portion of the day in Lugano before heading off on one of the multi-transfer routes I've outlined above and spending the night before your Bernina Express trip in Tirano.