If traveling by train, I second the suggestion of Foix as an easy, low-key daytrip from Toulouse. The castle itself is small, but has a good interpretive museum with multilingual audio-guides. The town itself is compact, charming, affordable, and not very touristy. If you're interested in pre-history and cave art, you can take the same train line for another 10-15 minutes to Tarascon-sur-Ariege. The Parc de la Prehistoire has life-size models of the Niaux cave and other cave art from the region. I actually like it better than visiting the actual caves. It's a 5-10 minute taxi ride from the train station. Plus, it's in a gorgeous mountain setting.
There's plenty to do within Albi if you go by train. However, if you do have a car and really like Toulouse-Latrec, you could continue on to his family castle in Camjac (Chateau du Bosc). It's filled with his family's belongings, as well as drawings he made as a child on the walls.
BTW. There's a podcast called Join us in France created by two women who live in Toulouse (one American, one French) so they have several episodes on places to visit near Toulouse.