You won't have any border issues on a day trip to Mostar if you drive, if you have a US passport: stop at the border crossing, show your passport, perhaps show a rental car green card (insurance card, required to drive a rental car out of Croatia) and off you go. Not saying you need to drive if just going to Mostar, though it would allow you to make some other stops (like the Dervish house at Blagaj which is only a short detour from the drive to Mostar). And the hour delay for the bus mentioned above sounds unusual, not the norm.
You could encounter the same kind of delay at the Montenegro border, anyway.
I would say they are very different day trips. Mostar is indeed unique and quite different from what you see in Croatia proper - the mosques sure give it a different flavor from Croatia! Keep in mind that the city is half Croat - the side that still dominates the town, as I understand it. My Sarajevo tour guide (a secular Muslim) suggested that many in Bosnia (perhaps Muslims) consider Mostar part of Croatia.
Mostar gets a lot of tourists now especially on day trips, FYI. I haven't been there since 2009 - understand it can be crowded in the day in the old town, which isn't really that big. The Stari Most bridge (a reconstruction of the one destroyed during the war) is amazing and unique. If you go to Mostar, dig up some footage on YouTube shot during the war showing what it was like there when the snipers were out - scary stuff, hard to imagine it wasn't that long ago.
By comparison, the town of Kotor itself was not a highlight of my brief visit to Montenegro. The town is nice - great old buildings and squares, some beautiful churches, etc. But it wasn't outstanding compared to say Dubrovnik. On the other hand, the scenery around the Bay of Kotor is amazing and unforgettable. Although one can take a day trip by bus to Kotor itself, to me that would miss the point which is the scenery; I was glad I had my own car because I could stop and take pictures, which I did numerous times. I wouldn't have wanted to try to enjoy the scenery through a bus window driving by!
There are some interesting stops between Dubrovnik and Kotor such as the lovely town of Pearst - beautiful little town, try to stop for a little while if you can. I think you can take a boat to the island church from here (I didn't). I spent a night in Kotor but think I would have preferred staying in little Perast instead.
The walk along the "walls" above Kotor provides more breathtaking scenery. The "walls" are actually a steep climb up many steps to an incredible viewpoint - cruise ship groups do it routinely, so you probably can too even though it's steep. The picture of the Bay of Kotor from the top with Kotor in the foreground will probably be one of your trip highlight photos. If you have a car, you can get more great views driving up above Kotor through the mountains, driving to the town of Cetinje, a windy drive with many hairpin turns (but places to pull over for views). Doing that drive to Cetinje in addition to Kotor etc. is kind of a lot for a day trip, though, although I think Rick Steves suggests it can be done in his book. (Could be done but it would make for one very long day.)
So to sum up, I'd say Kotor is about beautiful scenery more than anything else, though Kotor may not seem a whole lot different from Dubrovnik on the surface.. Mostar is about a different cultural and historical experience than Croatia, more of a contrast than Kotor is. If I knew it was going to be a rainy day, I'd lean toward Mostar - I don't mind exploring a town in the rain, but it can kind of ruin scenic views like those along the Bay of Kotor.
