Delft, Leiden, Haarlem and/or Hague for Amsterdam side trip?

There's no right or wrong answer. Haarlem is closest and is absolutely worth spending a day in if you're in Amsterdam. Of those four cities I loved Delft the most, but they're all terrific places, and wherever you go, you'll find plenty to see and do to fill your empty day.

You can't go wrong with whatever you choose.
The Hague has good museums (Mauritshuis, Gemeentemuseum and Panorama Mesdag) and the elegant area around the government complex of the Binnenhof.
Delft is possibly the most picturesque of this quartet, and has the Oude and Nieuwe Kerk (Old and New Church) which are both very much worth a visit.
Haarlem is beautifully situated on the river, and has the Frans Hals and Teylers museums.
Leiden has the largest historical centre of the four, and is at its best on Saturdays with its market behind the Town Hall.

We combined Delft and the Hague in one daytrip from Amsterdam. (They are so very different and it was an interesting contrast.) They are so close so it's easy to combine the two cities, although combining two in a day made us not able to see as much, especially in the Hague. Delft is so beautiful and picturesque and small enough to see most things in a half day. We felt content with only a half day in Delft.

I think Haarlem, Leiden and Delft are all good choices. I haven't been to the Hague so I can't speak about it. I would pick Delft, but that's me. As others have said, you can't go wrong!

You can't go wrong with any of these cities, and though I enjoyed them all, I liked Den Haag the least. Haarlem and Leiden are my top two choices.

If it affects your choice, Leiden and The Hague are the two where you cannot see everything in one full day out. I mean that as a plus. Do you understand that The Hague and Delft are adjacent? But if you combine them, you'll see even less of Den Haag. I notice that you didn't mention a windmill historic site. Is that another day? I ask because, for example, Kinderdijk could be combined with Rotterdam or The Hague, if you get an early start. Rotterdam might be even bigger than The Hague.

My vote is for Delft. It doesn't have a windmill but it has a charming little white painted drawbridge along a small riverway.

Delft no windmill? A few hundred meters from the railway station they've got an impressive classic one: De Roos.

I like Delft and The Hague. You can walk around the old town square of Delft and also go tour the Royal Delft factory (https://www.royaldelft.com/en_gb/)

It is a bit long, but very interesting streetcar ride from Delft to The Hague. I would recommend touring Mauritshuis, one of my favorite museums. It has great Dutch art and I love the building with the mix of new modern entrance/shop/café in the basement with the old world art building above. https://www.mauritshuis.nl/en/

Leiden and The Hague are outstanding places to visit.
I give Leiden the most charming award with it's flower decorated bridges over it's many pretty canals.
The Hague is a small city but so approachable from the moment you step off the train. The Parliament buildings and Mauritshaus Museum were our highlights.
Delft is smaller and the historic churches- Old and New, are very interesting.