Interested in history, architecture, museums, walking, general vibe
Amsterdam has all this in amounts far greater than 4 days can cover and to a greater extent than any other Dutch city. And Amsterdam has a ton of nice places to eat and stay.
My usual grouping of Anne Frank House, Dutch Resistance museum, and Corrie Ten Boom house are spread out enough to give you a few different areas to see (two in Amsterdam, one in Haarlem), and really only the AFH is crowded (on the outside area). Haarlem was a lovely day trip, but compared to all the offerings in Amsterdam, it wouldn't be my suggestion for a first (and shortish) visit to the Netherlands.
You can definitely avoid "massive" crowds by not spending much time in the more "core" touristy areas (say from Dam Square down through the Red Light District and ending by Centraal station).
But I wouldn't avoid those areas altogether - just choose maybe less "peak" hours to wander through? Amsterdam is a living city, so there is always activity - locals and tourists - but you can also pop out into parks and even the "countryside" with relative ease.
If you are going for a very RELAXING break, though, Leiden for sure was a notch down in activity, but also full of options for exploring, as was Delft. Haarlem, Delft, & Leiden would all be relatively near Den Hague, Rotterdam, and Amsterdam, so you could stay & relax in those lower key spots and then simply make a day trip or two out to the bigger cities if you wanted a bit more action.
Honestly, with so much to see and do, all the Dutch cities will work as a home base and have lots of options for fun filled days.