First off, good on you and good plan. Zagreb and Sarajevo are both great, overlooked cities. You’ll enjoy spending time there. FWIW, I think the time spent in Ljubljana on the tour itinerary is depressingly short. If you thinking three nights up here, I’d lean towards two in Zagreb and an extra in Ljubljana (though three nights in Zagreb would go down very easy imo.)
Zagreb seems to be the airport hub of the region, perhaps by default, but generally the best connections and price end up there. Bus connections to Ljubljana are frequent and easy, the train was rough.
From Dubrovnik to Sarajevo, if there’s no quick flights, I would look into a private transfer with a tour company. This would include some interesting stops along the way, and give you a good look at BiH. FunkyTours has a good reputation here, and I was very pleased with them overall (though not this itinerary.) A straight, long bus ride would work here too.
From Sarajevo, the easy answer is it’s your departure city on an open jaw ticket. Every flight from the US will include a layover/change of planes in Western Europe somewhere, and it’s worth seeing what the cost/convenience equates to. Croatia Airlines runs turboprop flights to Zagreb that take under an hour and are cheap, though I’m not sure of availability in October. Worst case is a long bus ride, or longer private transfer with stops along the way. (I was trying to arrange this myself last summer with FunkyTours. The stops looked very interesting and beautiful, but I couldn’t justify it solo. It’s worth considering.)