Here's the problem: "day trip"
If you are trying to avoid crowds (and from all indications, that's a very real issue at CT these days), the worst of the problem is because of "day trippers." Popular places are most crowded between 10 am and 4 pm, "Day Tripper" hours.
If you stay elsewhere and day-trip to CT, you will experience CT at it's absolute worst - peak crowding. Conversely, spend the night there, and you get to enjoy it in early morning and late afternoon/evening, when the crowds thin out and the place retains its charm.
One potential way to salvage your marital issue: arrive in CT mid-afternoon, and watch the crowds start to drain away. Stay one night. That way you get to enjoy a few good hours the first afternoon and evening, get up early and enjoy a few more hours the next morning, then flee by 10 or 11 am, head for that beautiful Tuscan villa, and spend your second night there.
Everybody wins (or, depending on your spouse, everybody loses...can't help you with that). Good luck.