We took our son, daughter-in-law, 3-year-old grandson, and 6-month-old granddaughter to Puglia in Italy for 2 weeks. The 3-year-old actually had his 4th birthday while we were there. He is now just turned 7 and he does remember things about Italy. But we did not take him or his sister there for their memories --- we took them for ours.
It was such a pleasure to share with him so many of the things we love about Italy, like gelato every day, car-free piazze where kids can play, great food (he ate everything in those days, including Italian cured meats), interesting old buildings, roof-top terraces, cars and other vehicles you don't see in America, and a multitude of things that are unlike from what he knew at home --- he was enthralled by all the different ways the toilets flushed, for instance. He learned some Italian words and phrases, and he lapped up all the admiration and attention that he and his sister got from people. And he still asks us how to say something-or-other in Italian, how do they do something-or-other in Italy, and when can he go back.
Our granddaughter ate her first solid food while we were there, a slice of fresh peach, and "starving wolf" does does begin to describe her reaction to that. I guess she could have experienced that at home, but it made us happy when that very first taste was eating something in Italy.
We have memories and photos of all of it, and I wish we could take them some place every year.
There is lots of information on these forums and elsewhere online about taking young children on such trips, but I'll just say that we stayed in two apartments in Puglia for a week each, so everybody had their own room. We went to no churches, museums, or art galleries except for the train museum in Lecce. It was late March/early April, but we went to some beaches (mostly so he could throw stones into the water), explored around in the old centers of several towns which were often pedestrian-only, walked in the countryside, shopped for Easter stuff, hung out in the piazza after dark, and just enjoyed being together.
If your child is like our two sons were at 3 or 4, and not like our grandson, you probably could go to more adult places like art galleries, just not for hours on end. How your child naps/sleeps or needs daytime naps to function is also a thing to consider --- our kids were just naturally easy and flexible about sleep (we take no credit for this) and were not bothered by jet-lag, whereas grandson, let's just say, needed extra planning and patience on our part. Also, of course, it helped with the trip to have four adults wrangling the two kids (and all their equipment!) instead of just two.