I think your planning is seriously ill considered, and you are underestimating the time required to do what you want to do.
For example : Your first travel day- figure at least an hour to get to the rental agency and pick up the car. Then it's a minimum 90 minutes to Schwangau. Except you'll likely be driving in morning rush hour, then you have to park and get to the ticket office. So more likely 2-2 1/2 hours. Then there's the wait to get into the first tour, even if you have a timed ticket (look this up online). Then the second castle. Somewhere in there you need to eat lunch. That's about 6-7 hours spent. Then the drive to Lucerne. You're kidding yourself if you think you can do it in 3 1/2 hours. You'll need to stop to get a vignette for the car when entering Switzerland, and at the end of the trip you'll have to hunt for your hotel and find parking. Count yourself lucky if it only takes 4 1/2 hours. By then it will be time for a late dinner and you'll be exhausted.
Driving to Paris- again underestimating driving times and not accounting for rest stops, getting gas and negotiating unfamiliar French cities (and I really hope you aren't considering dropping off in central Paris.)
Which brings us to the drop off - have you looked at the cost of renting a car in Germany and dropping off in France? The sticker shock wasn't a shock to you?? It's certainly a deterrent to many.
Oh, and don't forget that you will need to get an International Drivers Permit for each driver before you leave home. Make sure you have cash for the toll booths, since many won't take American credit cards.
OR
You could stay an extra day in Munich and spend it doing a side trip to the castles , then fly to Paris and have a couple of extra days there. Save Switzerland for your next trip, when you can plan more time to actually explore and enjoy it instead of having a few rushed glimpses. JMO