I stayed their for a few nites last summer. It was ok, no complaints, but if I was going back to the area I'd probably try somewhere else, Although this isn't to say there is something wrong with Vannes or that it wouldn't be a good base.
There is an old-town area to visit and the castle grounds are nicely maintained. At night, though, this area was pretty quiet. In fact the whole town seemed quiet at nite. I was staying on a hotel near a larger road and during the day there was a lot of traffic and I was wondering if I'd be disturbed by the road noise at night, but at 7pm or so it was as if someone flipped a switch. There were literally no cars out at all! Perhaps the norm is for people in smaller towns to be home at nite. The old town was pretty dead in the evening; a few restaurants were open, but you didn't see people walking around the area; personally it felt a little spooky to be walking down narrow empty streets alone at night. The action is down at the harbor, with a number of restaurants facing the water with outdoor seating and those were busy. Vannes is not actually on the gulf, though, from the harbor a channel leads eventually to a mini-bay (see a map online).
So if I was going again I'd look for a town that is actually on the bay, but that's just my preference Speaking of which, you should rent the movie "Monsieur Hulot's holiday" which apparently is something of a classic in France. See http://www.theguardian.com/travel/2009/jul/06/france-beach-holiday-short-breaks