Yes, uncomfortable for sure!! I'm tall, and still uncomfortable for me. We have the expanded passport (the one with more pages, because we too quickly blew thru our old ones) and it is like (I exaggerate) carrying an unabridged dictionary around one's neck....but seriously, there is a big difference in the bulk factor.
When weather is cool, I wear the neck wallet in front, but hide the bulges (points from the passport) with a scarf. In the warm weather (I like to wear long-sleeve t-shirts), I find very, very light weight scarves....or if wearing a dress, I will do like other posters have stated, and wear it with pouch on my back....still uncomfortable..........but less uncomfortable than losing a passport.
For my spouse, he wears the neck style, but we also altered an RS neck wallet money belt to be holster style, and that is where the large bill cash, credit cards, emergency contact info, and supply of US cash goes. Yes, it gets grossly wet in the hot weather, but a small sandwich bag keeps contents dry. We also carry an extra one, so if he's saturated from day touring, a fresh one can go on while the other is washed and dries.
My daily cash goes in a zip section of my travel purse, which is worn cross-style, kept in front of me with the zipper facing my body.
My sister, an excellent seamstress, fashions extra zip pockets into her husband's travel pants and her shorts/skirts, but they don't have the bulky extra thick passports....but that could still be more comfortable than having stuff hanging around your neck (and it does get heavy sometimes).
There is no right or wrong way........it's just a matter of what works for you and what is more comfortable than not.
In today's world, we just do not feel comfortable being without our passports on our bodies......just in case.
We also carry a color photocopy of the other's passport in our money belts.