I'm not sure that I can say that the places I'm going to name were "quieter than expected" in a positive way, but instead, they met our sweet spot of having a location with things we enjoyed doing in the area, and a relatively lively town life, with good people-watching, but not swamped with tourists. We realized that for us, however, we prefer locations that have some number of tourists. Their presence supports more restaurants and cafes, which we like.
We've really enjoyed Ghent, Belgium; Espelette, France; Bayonne, France; Arezzo, Italy (and were luckily there on the opening day of the monthly antiques faire, which I loved); Modena, Italy. We visited one palazzo near Modena that was interesting to us, but so little visited that a museum guard followed us from room to room, rather than having a guard in each room. By the end, we felt like he was another family member! I like Canterbury, UK, as well.
Evora and Tomar, Portugal were lovely and we were glad we visited, but a little too quiet for us (we spent three nights in each place).
Narbonne, France, which i researched and chose as a destination, was a little too quiet. We did enjoy sitting at their cafes on nice days, along with the locals. Also, Pamplona, Spain (not during running of the bulls); it just didn't click with us.