The SNCF website isn't working for me at the moment. You can try hitting https://en.oui.sncf. If it misbehaves for you as well, you can use the Deutsche Bahn website for your initial research. You just won't be able to get information on French fares there.
What I'm seeing at the moment for the trip to Brive-la-Gaillarde is that most departures are from the Gare Montparnasse and take 5 to 7 hours with at least one change along the way. Do pay attention to the transfer time allowed if you have to switch trains somewhere. Check the size (population) of the place you'd be changing to get an idea of how large and busy the station might be. It doesn't take long to change trains, but you sometimes have to go down a flight of steps and then back up another flight, and it's not something I'd want to do with a 5-minute transfer time unless I was wearing all my luggage on my back (which is not going to happen)--and even then I'd be aware a missed connection was possible.
There is, however, a lovely, direct (no-change) train that's the fastest overall. The possible disadvantage is that the train departs Paris at 4:41 PM and arrives in Brive at 9 PM. There's no small-town French rental-car agency open at 9 PM, so this may be a deal-breaker. It could also be an issue with picking up the key for the house you're renting. Otherwise, I think this would be grand. You'd have much of the day in Paris. You'd be able to pick up food for dinner (ideally not something smelly like fish), which you could eat on the train.
Another advantage of the 4:41 PM departure is that it's an Intercity train rather than a TGV. It probably moves a little slower down the track, but it makes up for that by not requiring you to cool your heels at one or more transfer points along the way. The ticket would probably be a good bit cheaper than one of the connecting routes involving a long TGV segment.
At the moment the DB website doesn't show any Paris-Brive connections arriving before 4:56 PM; even that might make you too late to get to the rental-car pick-up point that day. Maybe more trains will be added as things open up more. Otherwise, you might consider taking that late IC and crashing for the night in Brive. The historic center is fun to walk around, assuming you'll have some decent light in that part of town after 9 PM. I'm usually back in my hotel before then, so I can't tell you what to expect.
This paragraph edited: Another possibility: Pick up the car in Perigueux rather than Brive. There is an earlier train arrival in Perigueux (nice town). ViaMichelin suggests that it might take an extra 15 to 20 minutes to drive to Sarlat from Perigueux instead of from Brive. I have no idea where the respective car-rental agencies are located or what sort of traffic you'd face heading out of town. But the train trip will be just 4 hours. That early train leaves Paris at 9:45 AM and arrives in Perigueux at 1:45 PM. (Again, you can save time if you pack a lunch to eat on the train.) There's a change in Bordeaux with a 28-minute layover, which should be fine. The Paris-Bordeaux leg is a TGV, so it will probably be rather pricey.
If you decide to hop over to Rocamadour, be sure to allow youselves enough time to go up to the religious section of town. If all you do is walk down on the lower level, you'll see nothing but tacky souvenir shops and restaurants packed to the gills with tourists.
