Germany, Austria & Switzerland Itinerary

Fussen is near Garmisch and there is plenty to do in that area. Oberamergau is worth a visit and taking the cable car up to the Zugspitz.

There will be a hefty fee for renting the car in Germany and dropping it off in another country. If that is a concern, then consider dropping it off in Germany and then relying on rail transport into and within Switzerland.

I don't know what your interests are but if I were going from southern Germany to Grindlewald, I would consider staying a night or two in Appenzell, Switzerland. It's a rural area of Switzerland with lots of natural beauty, hiking options, small villages, and a folk museum. Appenzell to Grindlewald is about 4.5 hours by train.

DevN2 asks; Is it worth staying in Fussen for 2 nights? Is there much else to do around there, or should we consider staying in another location nearby such as Mittenwald/Reutte?

You might consider staying in Mittenwald or Reutte and doing Neuschwanstein, Hohenschwangau and Wieskirche one day and Linderhof another day. So, yes, you could almost plan to spend 3 nights in the area. If you go to Wieskirche be sure to go inside the church because that is the reason for visiting it. For the best view of Neuschwanstein take the bus from Fussen to Marienbrucke.

Most of your destinations are very similar. Why so many mountain towns?

"Following this we will be going to Fussen for 2 nights to see the castles..."

With 15 nights altogether, you could use these 2 nights and 1-2 additional borrowed nights to add a little diversity and visit "the other" southern Germany to the west, which isn't far off your route to Switzerland.

Burg Hohenzollern is far less inundated by tourists than Neuschwanstein (which btw is a late 19th-century palace, not a castle.)
Meersburg Castle on Lake Constance and Meersburg itself are worthwhile, as is nearby Lindau. Tübingen is a handsome university town on the Neckar River with an interesting historic town center.