I would choose Albi in the first place for the city itself (the cathedral of course but not only) and also for the possibility of driving in the "Gorges du Tarn" towards Ambialet.
https://goo.gl/maps/Bhfy8jtmYvSPNkfz5
The nearby town of Gaillac is lovely too (and the region is renowned for its vineyards)
The town of Rodez itself is relatively uninteresting but all the surroundings in "Les causses" and the Aveyron towards Espalion, Laguiole are worth the visit.
Castres is also nice with its old merchants' houses on the banks of the Agout river. And also its surroundings, La Montagne Noire (the Black Mountain), the village of "Soreze" and the lake of Saint Ferréol, the village "Les Cammazes"
https://www.tourisme-tarn.com/uk/discover-the-tarn/unmissable-attractions/castres-and-montagne-noire/
I would drop Montauban.
A good restaurant in Albi:
"La Table du Sommelier"
(there is also one in Gaillac)
http://www.latabledusommelier.com/restaurant-albi/index.html
Another near Gaillac:
"Le Montanais" in a splendid setting in the middle of the vineyards (of Gaillac wine, of course)
https://goo.gl/maps/9b167Jssd8g5TNWj8
Near Rodez, if you want to eat in an authentic farm:
"La Ferme Carles" lunch only, reservation essential
https://goo.gl/maps/Xx6XAJYSgVs6KK668