This question has come up before and I always see the same answers... don't worry about it... Well, this time I'm speaking up.
10 years ago, I reluctantly left my passport with the hotel front desk one night and come morning, mine and several others were gone.
Getting a new passport was such a hassle and took so much time. I had to hunt up the US consulate, fill out all sorts of paperwork and be interrogated. I did have to alter my plans, since I didn't have a passport and all, I was stranded for several days. I also incurred the expense of new pictures and passport fees. Eventually, I was issued a temporary passport that was only good for a year.
A year later I had to go through all that again back home and they eventually extended my passport for another nine years. And all that was the least of it... because my identity was stolen. It took me three years to deal with the ramifications of that.
So, I never let anyone keep my passport, I guess I'm just funny that way. On my most recent European trip, I scanned the key pages of my passport and printed and laminated it. One place in northern Italy asked to keep my passport and, with it already in his hands, he simply slid it into the key slot next to the desk well within reach of anyone passing by.
I took my passport back and asked him nicely to get any information he may need now or as an alternative I offered him my little card. I figured if that was stolen, I risk my identity being stolen again, but at least I'd still have my passport and could still move about and get home.
I was told by the consulate guys, that it would have been easier had I stopped by when I first arrived and "registered" my passport, so that's an option you may want to consider. I was also reminded that in the passport, it clearly says keep the passport in your possession at all times.
The odds are that nothing like this will happen to you. But, if it does, you'll be so sorry.
Cheers,
Maggie