Yes it makes sense to carry a small bag. Key is small! You don't need a ton of stuff, but a few essentials. To me that's camera, phone and charger, small baggie with Advil and tums, and scarf or sweater or sunscreen if the weather calls for it. My guidebook is an ebook on phone or else I'd have that too.
Water fountains - yes, common. Especially in Rome. Many of the historic fountains are still used for drinking water so fill up!
Phone charger - a small power bank is great. Charge it at night and then it can charge your phone during the day as needed. Mine is called the jackery mini but there are a lot of brands. I use my phone as my map (offline maps app) and my husband uses his as his only camera. So battery goes fast. This a non-negotiable essential to me and it's smaller than a pack of cards. Closer to a lipstick in size.
Some museums do make you check all bags and some permit smaller bags. Many allow shoulder bags and not backpacks. Carry a backpack under your arm in these situations or better yet just use a messenger bag. I feel strongly about this (for me, I don't care what others do but I will share my reasons in case they help you). Daypack backpacks are cumbersome. You have to stop every time you need to get something, while with a messenger bag you can access your water or camera while walking. Backpacks are also harder to keep an eye on and present a pickpocket risk. A shoulder bag can easily be carried to your front. Yes, you can put a backpack on your front but then you look silly and like you are protecting something of value. Finally, sometimes I use a backpack luggage - so how would I carry my luggage backpack AND my day backpack in the airport/travel days? Luggage on back, messenger bag over shoulder!