In March we took our two grandchildren to Italy for 2 weeks (and their parents, too!). Ages 5 months and 3-almost-4 years.
We took a mountain of stuff (some of which I thought was unnecessary. e.g. a folding stepstool) including 2 carseats, a stroller, a baby front-pack, a travel cot (with sheets), the 3-year-old's pillow and blanket, and all the baby's disposable diapers. It did take all four adults to carry all of this. However, we had a wonderful time and are glad we did it!
We stayed in two places (airbnb apartments in Puglia) for 6 nights each and then 2 nights in Rome before the flight home. We had to rent a 9-passenger van, but you won't have to do that. We did day trips of no more than a hour's drive away almost every day, which was actually fine because the 3-year-old needed naps but could only nap in the car. We pretty much only did one or two things every day. We went to no museums, art galleries, or churches, except for a train museum. We did a cooking lesson at a farm, but our grandson was free to run around with me and look at the cows, chickens, dogs, etc. We didn't do anything (such as a tour or a boat trip or a concert) that would not allow us to escape easily if necessary.
The kids were absolutely great in restaurants, although our son and daughter-in-law were nervous the entire time that they wouldn't be. Pretty much everything we did was planned around the kids except for one ceramics shop (which made all of us nervous, but again, was fine). We walked around the historic centers of towns, looked at boats and the little Italian trucks called Ape, threw pebbles into the water at a pebble beach, visited farmers markets, ate gelato, ran around in every piazza, etc. Our grandson was enchanted by all the things that were not like home, such as hanging up the laundry outdoors and seeing how different toilets got flushed and all the pedestrian-only streets.
My guess is that our grandson (the 3-year-old) would have been fine with 4 nights in one place. He did have trouble sleeping the whole two weeks, but he has trouble sleeping at his own house. I could not tell if either child had jet-lag during the day, but I've heard that kids do have it for a couple of days and are awake at night.
In general, I would say that what was hard for the kids (especially our grandson, the older kid) was not moving too fast but moving too slow --- that is, all the times when they were stuck in airport lines, waiting for trains, waiting at the car rental place, waiting while the adults got Italian SIM cards for our phones, waiting while the adults packed and unpacked, etc etc. Travel involves a lot of standing around! Of course, we had ways of keeping the kids occupied and not too cooped-up, but still, anything you can do to minimize such times will help.
I should mention that, unlike our two sons (who we traveled with a lot), our grandson is not real patient or self-entertaining, so your baby and toddler might be different. Our granddaughter, now exactly one year old, is both patient and self-entertaining, but also really needs to run around and explore and do active things, so I imagine that a trip with her now would be a whole 'nother adventure!
Our son and daughter-in-law were definitely surprised that the kids turned out to be such good travelers and could be so flexible about their daily routines, especially napping. You never know until you try it! A lot of pre-trip strategizing went into this trip we took, so I'm glad that you are planning your own trip now and thinking it through --- it will really pay off.