Southern England 2 nights

I am looking for advice on a town to stay in for 2 nights that is not too large or too touristy and makes sense as a stopover between Heathrow and Portsmouth. We will arrive in Heathrow noon on a Monday in June and depart on a Brittany Ferry two days later at 8 pm from Portsmouth. We will not have a car. We are a young couple who travel light and like good food, farm shops, countryside walks, and medieval sights. We will spend a week touring Cornwall to London by car later in the month so southwestern England will get explored then. Open to any advice on towns, accommodations, restaurants, transportation, etc.