Lots of really excellent advice here, so I will only add one thing: GO EARLY. Whatever day you pick, GO EARLY. Did I mention that you should GO EARLY!? ;-)
I live in Chelsea/Fulham and it’s a haul for me to get to Borough Market, but I suck it up and get up early enough to rock up exactly at 10:00 on a weekday (it opens at 9:00am on Saturday) so as to avoid the worst of the crowds. There’s a lot to see and a lot to eat, and it’s much more enjoyable when you’re not jammed in like a sardine.
Tips: Southwark Cathedral is beautiful and is right there if you have time to pop in. And there’s a Shakespeare connection!
If you like oysters, Richard Haward’s is the place: prices are reasonable, and the oysters are lovely.
The George, a gorgeous 17th century coaching inn, is about 4 minutes’ walk away (75 Borough High Street). Unless you like Guinness (theirs is mediocre) this is a lovely spot for a relaxing pint after the bustle of the market. It’s crammed with charm.
Have fun!