"Pretty sure the Prime Minister isn't going to invite me in..." depends what donation to party funds you offer (and this is not a joke!)
start with the easier bit: both Inns of Temple do tours, if you can produce evidence of ancestors it might bump you up any waiitng list!
http://middletemplehall.org.uk/tours-of-middle-temple-hall.html
https://www.innertemple.org.uk/estate-garden/guided-tours-of-the-inn/
I think some of the streets in Middle Temple are open to the public on foot (well at least one is signposted as a route so they'd better be!).
Downing Street is difficult, there are pretty strict about security, you can't get within 300ft of the front door. I think from the Whitehall Gate you can just about get a glimpse of the front door but it would be best show your passport and explain to the police BEFORE you start poking cameras through the gates. But they let the press closer and they are generally scum (that wasn't a joke either....) so nothing is impossible. How good is your genealogical documentation???
By the way, the street is named after the first Sir George Downing, there were 3 in total and all will need several more greats in front of uncle.