Villages/towns outside of Paris

Three I would recommend:
1) A day trip to Normandy to see the D-Day sights & Bayeux Tapestry

2) Giverny to visit Monet's Gardens (one of my favorites)
3) Fountain Bleu to see Napoleon's Palace

You could spent that time in Bayeux, Normandy near D-Day sites and museums and home to famous Bayeux Tapestry c.1066.
Or take a train to Dijon in Burgundy and switch to Beaune. Visit wine villages nearby.
Go to Versailles on your own using RS Guide.

On my last trip to Paris Oct 2019 I finally made it out to Malmaison (in the suburbs beyond La Defense.) It is a lovely smaller estate which was the true home of Napoleon and Josephine as they chose it themselves. It is beautiful and the gardens are quite lovely.

Chateau Vaux le Vicomte; Senlis (medieval village reached by train/bus on one ticket; Reims for the Champagne houses; Provins (another medieval village- walled and they have hawk exhibitions and such in summer). The villages alone the Marne which has lovely footpaths along the river. Get ahold of Simms' book 'an hour from Paris for about 20 day trips to less well known spots. I particularly remember one we did to Conflans St. Honorine with a walk along the river and lunch on an island:
https://janettravels.wordpress.com/2015/11/18/conflans-sainte-honorine-easy-day-trip-from-paris-with-big-payoff/
Crecy la Chapelle -- sort of a little Venice with wash house and canals and a place where a number of well known artists painted. Moret sur Loing, charming medieval center and then walk along the Loing to its confluence with the Seine. Just lots of great options in this book.

Auvers sur Oise where Van Gogh ended his life is a wonderful day trip from Paris. So much to choose from.

If you did the Rick Steves tour Paris and Heart of France you stayed in Bayeux for the Normandy area and also caught Giverny on the way back to Paris.

Depending on the days of the week you are in Paris for your first 2 nights/1 full day you could do Versailles then unless it's a closure day. Versailles is easy to do on your own if you are comfortable buying a ticket for the RER train to the station there.

On your return nights you'll have 2 full days so on one of those days you could consider some of the other suggestions. I also enjoyed Chantilly as a day trip from Paris. Rick tells how to do it in his guide book.

I would combine Château de Chantilly with Senlis, the latter to reach by bus. Château de Vaux-le-Vicomte is fantastic too, taxi or Chateaubus from Melun railway station.
https://chateaudechantilly.fr/en/access/
https://vaux-le-vicomte.com/en/prepare-your-visit/access-transport/

Recommend Fontainbleau for sure. Also, Senlis and Chantilly. We have done almost all of the spots visited and they are all worth it. One note of caution concerns Versailles. It seemed crowded with hordes and herds of people so as to render the experience less than pleasant. Normandy merits couple of days, but I'm not sure about train service to any of these places. Have you checked SNCF rail schedules? If you are not limited to trains, maybe a small tour group would save time and trouble, traveling in a van with driver/guide who knows the ropes.

Another vote for Senlis, Vaux-le-Vicomte, and Malmaison. We also really enjoyed the town of Sceaux which has a lovely chateau and grounds.

Lots of good suggestions here, but you really cheat yourselves if you do these all as a day trip from Paris. I stayed 2-3 nights in Chartres, Versailles, Fontainebleau, Reims and felt like even that time was too short. Most of these place have other things to see besides the on big tourist attraction.

I would also suggest Fontainebleau, all the more so if you're interested in Napoleon, take the tour of the Chateau. The Napoleon Museum is about a 30 mins walk from the Chateau. My other two suggestions are Amiens and Soissons.