Dublin to Belfast as a day trip is eminently doable if you plan to restrict yourself to exploring the beautiful city, visiting the Titanic Museum, etc.
Trying to include the Causeway Coast, the Hedges, etc. would be a bit of a stretch since they're located another 1.5 hrs north of Belfast. If that's your focus then I think Claudia's advice is on the money - ie head up to Belfast the day before and overnight there once you've collected your rental car. With an early start the next morning you can see what you want to see at a leisurely pace, maybe to include a visit to Derry - one of the most evocative cities in Europe before returning to Belfast for the train ride back to Dublin.
Time and interest permitting you could include a drive along the very pretty Antrim Coast - one of Ireland's most scenic drives on a sunny day.
Another thought: you could save yourself a step and some time by just picking up your rental car from Dublin airport on your way north, maybe driving up to the Causeway Coast ... staying overnight in Portrush maybe. Though you'd probably have to skip Belfast this trip (which would be a shame) it would certainly streamline your proposed itinerary.