legally, in most countries, you are supposed to carry ID at all times. Some countries it actually needs to be a passport, some will accept a paper copy, and I believe a few will take a driver's license. I always carry my driver's license because there is no way I'm leaving my passport for an audio guide. Lately, I'm running into more situations in which I need a passport, like verification at the Alhambra and with my Swiss Travel Pass in Switzerland. Even though most often legally required, if I'm checked into a hotel and I have no perceived need for a passport during the day, I leave it in a safe spot in my room. I don't want to carry it with me. It's just one more thing to think/worry about. Except in transit between cities/hotels, I don't use a money belt. If I'm driving a rental car, I'll have my passport with me along with my IDP and driver's license.
I think the important thing to remember, is that everyone should know the laws of the land where they are traveling as well as common practices. Evaluate your own situation and risk and make your own decision. I'm a white woman traveling with my husband and we are in our 50s, we don't look interesting. If I was younger or a minority, I might be more likely to carry it.