I can't speak for Chartres, but I had a very enjoyable day trip in Reims, taking the TGV from Paris for the day.
I loved the cathedral in Reims; the Chagall stained glass is amazing.
Of course, the highlight of the trip is the Champagne! Make sure you get appointments for the tours at the champagne houses - you only get to taste if you take the tour. I made my appointments by stopping at the visitor's center, right across the street from the cathedral; they called the champagne houses and made the appointments for me.
I went to Pommery and Mumm; I enjoyed them both, but the Pommery tour was better, in my opinion. If I had budgeted my time better, I could have fit a third house in, but I spent too much time wandering around the city and didn't make it to Tattinger in time for their tour.
One other recommendation, if you really like champagne - pay for all the tasting options! At Mumm, I was one of three people who had paid to taste all their champagnes, from their "regular" nonvintage to the special vintage releases; a large group of people who came in with a tour weren't able to taste the good stuff. I felt sorry for them...but not sorry enough to share my one glass with a dozen people!