We will be traveling from Paris to Reims to connect with the RS eastern France tour. Wonder if we need to book the train before we leave or can we book day of?
There are plenty of trains each day from Paris Est station to Reims. Even trains leaving tomorrow morning have seats available. While tickets bought weeks in advance for the TGV trains may be cheaper ( although with restrictions), even next day tickets don't look that expensive. However they could sell out on high traffic periods like holidays, commuter rush hours, or Friday/Sunday nights. . There's no advantage to buying TER tickets in advance, since they generally don't have reserved seats. Look on the SNCF website for your dates, and compare with those on the same date next week https://www.sncf-connect.com/en-en/home/search
You can book on the day itself. However you will pay full price, and if the train is sold out you may end up with a ticket without a guaranteed seat, and sitting on the stairs the whole trip (like I saw some people on the TGV to Quimper yesterday).
What I would suggest is do the following: Use www.sncf-connect.com to look up times, prices and availability for today, tomorrow, next week and next month. That will give you an idea what the typicla prices and availability is, and you can decide for yourself where the sweet point for price and convenience lies for you.
For Paris–Reims, booking the day before should be fine in most cases, but checking the schedule a few days ahead is still useful. That way you can see whether the train you want is a TGV or TER and avoid getting caught by a busy departure time.