Could you give me some insight into what you have taken along with
you?
I'm spoiled by the image quality of my (now mirrorless, formerly DSLR) interchangeable lens cameras. But I hate lugging heavy gear along on vacation trips. Especially when I'm walking around all day and thus aware of every ounce of stuff I'm carrying with me for the day.
I have a Sony RX100 iii (it's not the latest greatest version of that camera, but still on the market) which has been a handy travel camera for me. You can look up the specs: 24-70mm zoom 1.8 lens, pop-up viewfinder, 1" sensor. The latest version, mentioned above, is the RX100 vi, which has a much longer zoom at 200mm, but is way over the $300 mark. Not tiny but will fit in a coat pocket and certainly in a day bag.
That said, on recent trips to the UK, my favorite "travel" camera is my Sony A7R2 + 35mm f2.8 lens. Not as compact as the RX100 of course, but not too bulky, and I'm spoiled by the image quality and dynamic range of the sensor. That setup fits in my usual day bag and I love it for urban photography.
Because I found a great open box deal, last year I bought a Panasonic Lumix FZ1000, which although not tiny is a wonderful bridge camera. I've had some great results shooting around town with it. If I ever get as used to its controls as I am with the controls on my Sony cameras, the FZ-1000 will be a great travel camera. It's got that 400mm telephoto zoom reach and a 1" sensor - but it actually weighs a bit more than my A7R2 + 35mm lens combo.