I don't know anything about that castle. Olomouc itself is very picturesque.
Rome2Rio.com can be helpful in cases like this by advising as to whether you're likely to find rail and/or bus service covering part of all of the trip, though here it appears you'll probably need a taxi to finish the trip: https://www.rome2rio.com/map/Olomouc/Bouzov-Castle.
Understand that Rome2Rio has some pretty severe limitations. Experience has taught me not to trust the travel times for buses and trains, or their frequencies, or their fares. You need to drill down until you find a link to the appropriate company's website, where one hopes schedules will be available. Time permitting, I go by the tourist office in the town where I'm staying and check with the staffers about transportation options. On rare occasions there is local bus service Rome2Rio doesn't know about.
Here it appears the train gets you closer than a bus would, perhaps resulting in a slightly less expensive taxi ride, so if it's convenient I suggest stopping at the train station to verify the schedule with the experts.
Before you set out for the castle, Google for the telephone number of taxi companies serving the town where you run out of public transportation options in case there's no taxi waiting at the station when you arrive. I have done things like this without having a cellphone I could use to make a local phone call, but I go in knowing that it may cause me grief--though it never has, so far.