We are looking for a scenic, quaint town with a slower pace.
I am currently infatuated with Amersfoort. It's an absolutely lovely little town with an outsized bell tower and thick beautiful sections of city wall and gate still standing. It has the Mondrian museum, but is mostly just a pretty, upscale, not so touristy place to have a good meal and stroll around with the expensive prams etc. The two times I've been it seemed to me that locals were quite nice, more town people less overwhelmed by tourism. If I moved to the Netherlands it would be high on my live there list.
Delft is good too for what you describe!
My buddy and his Dutch wife and family love doing a day trip to Harlingen. It's north of Amsterdam on the North Sea and very much focused as such - pretty ships and salt canals, sturdy brick merchant row houses. She's Friesan so they bias north, but it's a cool town, a bit more Denmark-like than places in North and South Holland.
And of course my other true heart just south of the Green Heart, Den Bosch. It's bigger than the other places above but just an absolutely outstanding place for a tourism visit and great meal. Tons to do and see. The Hieronymous Bosch museum is wonderful. The boat tour of the city's mostly underground canal system is unique.
Amersfoort 1-1.5h one change in Amsterdam
Delft less than an hour direct or easy change in Leiden
Harlingen a bit of a trek, 2.5 hours on train and bus and bus, so probably not in your preferred window logistically.
Den Bosch 1.5h or less from Haarlem, one change in Amsterdam.