A car makes things easier, but it comes down to how many of the little tiny places you want to go to, and over how few days. There's bus service from Colmar to at least some of the Alsatian villages; you'll probably struggle to move from on to another of the smallest places directly. However, if the ones you want to see are close enough, a taxi might cover the leg at less cost than a rental car. That assumes, of course, that you can locate a taxi company working in the area; I've never looked for one, so I don't know how easy that would be.
There are probably more tiny places in Provence without public transportation than you'll find in Alsace, but again, how many of those places do you want to go to? Would you settle for a one-day bus tour going to a few of them?
I am always a bit reluctant to recommend the website Rome2Rio.com, because I've seen myself how wildly off-base it can be on travel times, frequencies, and fares. But if it says rail service exists, it almost certainly does, and if it says there's a bus, it usually provides the name of the bus company (often also a link to its website) so you can go straight to the source.
I'd start by reading a guidebook or two and making lists of my must-see towns and my like-to-see towns, then checking what sort of transportation is available to each one. Maybe it will turn out that all or nearly all the places you want to see have train or bus service. You really have to know your rough itinerary before making a decision about the best transportation.
I think the odds are good that Colmar would be your best Alsatian base if you don't have a car. Provence is trickier; I'm not sure there's one single hub with buses fanning out in all directions, though there might be.
I went to the Dordogne in 2019 and found I needed to set up shop in multiple different towns in order to have decent access to buses that would get me to some of the small places I wanted to go. I think Alsace is easier than the Dordogne; I'm not sure that's true of Provence. I had a good time Provence and in the Dordogne, but I definitely missed some stuff.