Yes, I'd rather have my passport when I need it. Which is why, on occasion, I chose to leave it at my hotel. Again, depending on the country and my assessment of the situation, I may be more concerned about loss or theft than I would be about leaving it in the room. If I take my passport with me and its lost or stolen, and I need to get on a cruise ship or fly home the next day, I'm out of luck.
I'm very cautious when I travel, but I'm also doing the navigation, carrying the snacks, taking pictures, admiring views, handling any tickets, pulling out cash and/or credit cards, chapstick, and sunscreen, and looking at the guidebook and my notes, all in a place I've probably never been before.