What would you rather do if you had to choose between those two?
Borough Market is staider and more middle-class foodie, Brick Lane has a big Indian/Pakistani flavour, although still with a sprinkling of hipsters.
They are very different so it depends what you like.
My preference is Borough ... but I love food markets and love to cook. Borough is primarily producers. Local produce, cheese, meats, seafood, baked goods, etc. It is a great place to go for lunch. I prefer Friday to Saturday as it can be absolutely packed on Saturday.
Brick Lane has a number of restaurants (a big concentration of Bengali restaurants). I've only been there on a Sunday when they have a market (although it is more of a flea market with a lot of misc stuff and some food). It is a short walk from Spitalfields Market to Brick Lane so you could combine these two if you go.