Best Base for 3 days in Provence

Well, you are gong to have a blast. So much to see, you really cannot go wrong.

Fodors has, or had, a book of walks in France that included several in Provence. Casis is feasible, by rail even, if it's a big enough priority to warrant your time.

To answer your specific question about base. With 3 or 4 nights you probably want just one base, to minimize logistical stuff. If that base is a smaller town such as St. Remy then you need a car the whole time. A car is also feasible in a small city such as Arles, so I;m not exactly saying that transportation dictates your choice of where to stay. Except a small town really needs wheels.

Many wonderful places and Roman antiquities are accessible by rail. Other places are doable by bus but it gets logistically more constrained that way and a car is helpful. Still other places, you just need a car or a bicycle or something. "Wandering and hiking all day ling" might fall into that category.

Have a blast!

Arles or St. Remy would work. Arles is probably better if you won't have a car. As for can't miss, we liked Arles, Pont du Gard and the town of Uzes, Orange (for the amphitheater), St. Remy, Gigondas, Gordes, Fontaine de Vaucluse, Les Baux de Provence, and the Roman ruins of Glanum. To see these sights a car is necessary. So then you're back to square one - Arles vs St. Remy. Both are great places to stay. We spent 10 days (5 each in Arles and St. Remy) in Provence and although we made it to the Camargue we did not have time for Cassis. You will need to be selective and plan good driving routes and you will be able to see some of the great sights of Provence. Whichever towns and sights you choose you really can't go wrong. Have a great time!

You can home base in Arles if you don't have a car, or in St. Remy if you have a car.
Arles, Nimes and Avigon are on the train route, you would explore them on foot.
Lodging in the historic center part of Arles is difficult with a car, not impossible but challenging.
Rent the car for 2 days, not 1.
Spend a day seeing Nimes and Avignon; take the train to these cities, and then tour the main sights on foot.
You could spend most of a day exploring just Arles.
Some travelers find Avignon to not be that interesting: the Papacy moved from Rome to Avignon for a number of decades in the 1300's. And there's the Bridge of Avignon, made famous in the French children's song.

I've stayed in Arles and St. Remy, both times with a car. I did a day trip to Cassis from Arles. I loved Cassis! I would consider having a car the entire time in Provence to maximize what you can see and do there. I can recommend a hotel in Arles with easy parking and in & out of town for drivers if you are interested. I stayed in an apartment in St. Remy.