Driving through the Neum corridor - the small section of Bosnia you drive through between Dubrovnik and Split - is routine. There are two border checks; if you have a valid US passport it's generally just a flash of your passport, quick and through, not a hassle. In theory they might check your rental car's insurance card, but I've never had mine checked there. (I have had it checked at other border crossings between Croatia, Bosnia, and Montenegro.)
Still, you're right - it's possible the rental car company may charge you extra just to drive through Neum, because technically you are leaving the EU. Each car company handles this differently. At least a few make exceptions just to drive through Neum and won't charge you more for that. Check with the rental company.
At worst, it's not really a "hassle" just an extra cost to rent the car. When you shop for cars, add it to the total cost and don't worry about it. Shouldn't be more than an extra 40 or 50 Euros total for the rental (if that). If saving that amount is worth taking a bus to you to avoid, maybe you should do it. Keep in mind that sometimes longer rentals have a lower average per-day cost than a shorter rental, anyway, so a longer rental from Dubrovnik may incur that cross-border fee but still not cost much more total. Do some comparison shopping. (Try EconomyCarRentals, Kemwel, and Auto Europe for example.) You can also email car companies directly and ask about a fee for Neum; some will respond to you in English.
