Frank II, I think you're right.
I'm a woman, so I probably don't get it, but I see no difference between a plain T-shirt and a Polo shirt, except that Polo shirts often seem to have some kind of emblem that obviously indicates a higher income level. Perhaps yours don't. I've always really disliked any kind of brand label on anything.
When my husband was traveling with me, he always brought dark T-shirts, both long and short-sleeved. Sometimes they had some kind of picture on the front. No sports teams or anything USA obvious. Still, I'm sure your friend would think he was a disrespectful American.
Wearing a well-designed black Joe Bonamassa T-shirt one morning at our hotel in Nafplio prompted a discussion with a German tourist who'd recently been to a concert. A favorite is black with a white design featuring the progression of a primate through the stages of walking upright to being a mechanic, working on a car on a lift. It always gets a smile, no matter who sees it.
The one time he dressed up for a dinner on a Seine River cruise, most of the men were not dressed up any fancier than a long-sleeve knit shirt or sweater of some kind and dark jeans. BTW, we heard no English spoken around us on that boat. He had to lug that jacket, shirt, tie and nice pants all over France. He said never again.
I haven't been to a restaurant, the theater, an opera or a concert anywhere that dressing up was expected for over 40 years. Of course, some people dress to the nines, but in my experience (TX, OH, OR, WA, AZ, Europe) they have been a distinct minority.