As someone who's been around the forum for a while now, I'd ask you both Olivia and Trelawney not to let it upset you too much.
It's just the way our cousins from across the pond think in a lot of cases. It's a question that is forefront on many people's mind because of how they're raised culturally. I don't want to get into a big sociological discussion of why that is. It's not just with London that safety comes up regularly.
I find one needs to cut people a little slack and not think it's a silly question when you're speaking to people with an attempt at grace on here. Particularly in the case of the OP here, as they have two young people in their care and they're looking out for them.
I dismissed the safety issue in one short line in my original reply because it wouldn't have even crossed my mind if it hadn't been mentioned. I know this area well enough to say that because I live in the neighbouring borough.
OP, it's a very safe area. People pay a lot to live around this area as Trelawney points out, as it's desirable.
I like to take a pragmatic view of safety in London. You can walk the streets anywhere. London is a city of around 10 million people according to census. Probably many, many more than that in reality.
Looking at you guys' location, I may have been around the inner London boroughs, including in the dead of night, more than than you. That's an assumption I'm making. I know that around Hackney, safety isn't a worry for me, but it's a consideration. It's going to be more so for women and young people. There's always going to be a dodgy person or two around. The likelihood of you meeting them on vacation is actually very slim, and they're probably not going to be engaging in behaviours that are actually dangerous. There's nothing to worry about in London if we put safety in perspective, but don't ignore it.