Both Google maps and viamichelin.com estimate the driving time at close to 5 hours. That's net (probably optimistic), without traffic and without looking for parking in Monreale. It sounds fine as long as you're not planning to sightsee in Agrigento that day. If you get to Monreale during mass, use the time to visit the cloisters first. You may also want to spend a little time wandering around the town (though maybe the shops will be closed on Sunday). At Segesta, you can also take the bus up to see the theatre - as long as you're there, it's well worth the extra time.
What you are missing, most of all, is visiting Selinunte. Of the three temple sights, I liked Selinunte best. Agrigento had considerably more visitors than the others (weekday), and a very good museum. My trip was at the end of April, carpets of wildflowers everywhere.