If you only have limited time in Belfast and want to get on the road, I would recommend stopping first at the Titanic Museum; this is one of the best museums you will visit anywhere. Or, take the Black Cab Tour (https://belfastblackcabtours.co.uk/ ) to give you a sobering overview of the Troubles, past and present. Belfast is worth 2-3 days, but to answer your question about whether you should stop there, even for a few hours, my opinion is YES.
We took a similar route that you are doing, only in reverse. We spent 5 days in Northern Ireland, and for your route I would recommend going a little past Bushmill's (5-6 miles) to Portrush (I am a huge golf fan and we stopped at Royal Portrush Golf Club, host to the British Open). Another highlight was taking the ferry at Ballycastle to Rathlin Island......about a 30 minute trip featuring lighthouses, puffins and beautiful scenery. Stops we made along the Antrim Coast included walking the Carrick-a-rede rope bridge, touring Giant's Causeway (a huge favorite), Dunluce Castle and a side drive through the Dark Hedges. The Coastal Causeway in total is a 1-1/2-2 hour drive from Belfast to the Antrim Coast.
We stayed at Bushmill's Inn which was perfect. While there, Bushmill's had an Orange Day parade; we also visited Bushmill's Distillery. I think you will really enjoy Bushmill's and it is a great place to base for your trip.
Good luck!
