Would you visit? And how long?

Rotterdam and The Hague both have good art museums. Do you have any interest in those? That would affect the time recommendation.

Agree with above poster. If you are wanting to explore the museums, it impacts how many days you'd want to visit. Last November I based myself for 5 nights in Rotterdam and, among other things, did day trips to The Hague and Kinderdijk. Don't feel like I was able to cover as many museums as I wanted to in Rotterdam or The Hague. Kinderdijk is entirely dependent on the weather. I lucked out and got about 2 hours of sunny weather before the clouds rolled in and it started raining. Had it stayed sunny, I could have easily spent another 2 hours there - great place to walk, bike and take photos, plus they have a small visitor center with a film about the windmills and you'll want to tour inside whichever windmills they happen to have open on the day of your visit.

Are you driving or taking the train?

Train

Kinderdijk is the best part of a day if you combine it with nearby much overlooked historical Dordrecht. They are both easily reached from Rotterdam by taking the Waterbus, a river ferry.
For me Delft is a half-day visit: the interesting part is rather small. Make it a whole day by combining it with Schiedam, 8 mins away and undeservedly almost totally forgotten. It has some huge windmills right on the edge of the historical centre, used for the gin industry which was once its major i
Haarlem is best done as a day trip from Amsterdam. It has two interesting museums: the Frans Hals and the Teylers, the latter being a late 18th-century gem, combining science with art.
The Hague is a whole day if you are an art lover. Besides the well-known Mauritshuis (Golden Age) and Gemeentemuseum (modern art), you'll find Escher and the Panorama Mesdag, being one of the very few remaining panoramas in the world. Weather permitting you can combine it with the beach at Scheveningen, a tram ride away (and a few stops from the Gemeentemuseum).