Hello from Wisconsin,
Olomouc can keep you happy for a couple of days. There is a most wonderful church, Basilika?, on a hill outside of town that is worth the bus ride to get to it. As for the size of the town. We never felt a need to ride the tram except from the train station into town and that was because of a light rain.
The train problem of Trebon to Olomouc is that most lines run from a place to and from Prague. So it wouldn't surprise me if you had to return from Trebon to Prague to get to Olomouc in the quickest fashion. And let me say that Brno is an alternative in itself. A bit more gentrified than Olomouc but still a good place to stay for a while.
A car rental from the airport might be an alternative. There is an express bus that connects train station and airport. TOnce you pick up the car there is a four lane highway just out from the airport entrance road that will swing you around the south side of Prague. Take the E50 to the E55, both major roads so the traffic is smooth and the roads simple and fast once you are out of the Prague traffic. Way under 3 hours drive time Prague to Trebon. I am not a fan of the superhighways in the Czech Republic any more than I am of our interstate, but if you want quick, they sure work. Once you are near Trebon you can take rural roads to see the beauty of the country. And from Trebon to Olomouc it is under 4 hours and it would have you drive by Telc (an absolutely lovely town, and a must stop) and then drive through Nove Mesto na Morave (no one has heard of that place, but it is the region where Czechs go for vacation and the countryside is drop dead gorgeous) before getting to Olomouc. A full day of auto touring. Driving any of this with a good map is not a problem, remember you don't drive in the left lane unless you are in the active act of passing and you always use your blinkers. On the 4 hour way back to Prague from Olomouc you can pass through Litomysl with its bow tie main square and Kutna Hora of Ossuary fame. A good driving break before going on the Prague to complete the route.
I would budget more than two days, and less in Prague. Prague is up to its ears in tourists in the tourist areas.
Another thought...do Trebon and Telc. You could do worse with limited time.