My bases were Dinan in the northeast, Lampaul-Giumiliau in the northwest and Vannes in the southeast. I ended up without a base in the southwest but visited Locronan and Concarneau and Pont Aven. If you want something bigger, you might love Concarneau. I just felt Quimper borders on being a large industrial port and too big for my taste.
I enjoyed Dinan but it's a pretty major drive from the west. Vannes was a great base for the prehistoric ruins in the area; plus it had more of a medieval center than I would have expected and a nice port right next to that. I also really liked Auray, a much smaller version of Vannes. I had many favorite places in Brittany, Vannes was my favorite base (but be walking distance to the port/old center).
Almost every place gets tourist traffic in the summer (though less than much of France) so there will be cafes and accomodations in virtually every town. I'd probably least recommend the northwest unless you really want to see the Parrish closes (unique to the area) or are determined to escape tourism. Towns I visited had relatively few cafes. I had to drive to reach scenic areas rather than just walk out the door. I preferred the look and feel of port towns, even though ports are often a ways upriver from the sea.