Intriguing question! I think it usually serves as a way of recognizing that the question or dilemma isn't so important in the grand scheme of things, even if it's really preoccupying to the person asking the question or discussing their dilemma. I don't usually think about it too much when I see the expression, but your post has really got me thinking, and I had two additional thoughts:
First, that travel as a hobby can have positive and negative impacts. Just like you said, local economies can really benefit from tourism. Individuals might become more aware of other ways of living and doing things, which is great too. On the flip side, there's an environmental impact to travel that isn't strictly necessary, as well as the various impacts of over-tourism - places being overwhelmed by tourists, or local apartments all being turned into short-term rentals, and on and on. Travel isn't necessarily a purely good thing.
The other thing that came to mind is that while I don't feel much negativity or even think about it much when someone says something is a "first world problem," I do dislike when people talk about places as being "third world countries." Sometimes people seem to focus on a "have or have-not" dichotomy that doesn't leave much room for nuance. It seems overly simplistic to think the world is split into countries that are modern/developed/rich and not modern/undeveloped/poor, when most places are actually somewhere "in the middle." There's such a wide range of lifestyles, quality of life, strength of economy, and on and on, but that tends to get erased when folks categorize a place as "third world."