Yes Giverny is mobbed in June, don't consider going without pre-tickets or you will waste at least an hour in the ticket line, maybe longer. And when you arrive step lively for the bus as it left many people standing at the stop when we were there and there was no second bus in sight. Same on return to the train station; get to the bus stop 15 minutes or so ahead so you can be sure of getting on the bus. They pack it full but still leave people behind.
We considered doing this trip but decided to do it on two days as we wanted more time in Giverny. I can see it could be done as Rouen historic center is quite small and you can see the major sites in a short period. It is a bit too rushed for our style of travel.