First question: are you renting a car, or planning to rely on public transport? Scotland has a robust network of bus routes as well as pretty good railroads, but it still takes more time to get here & there by bus or train than in a rental car. With a family of 4, you'll probably save $ renting a car instead of buying tickets for 4 people.
Second, I see the Highland Games falls right in the middle of your trip, so you actually have to return to Glasgow twice. You couldn't adjust your time frame to arrive in Glasgow around 5/12 or depart for home around 5/16, could you? And/or fly into Glasgow and out of Edinburgh?
The places you want to visit are scattered hither & yon. I'm guessing the Isle of Muck is your top priority, given the family connection. It will take you, realistically, 2 days from Glasgow to get there. I trust you've researched the availability of ferry connections and the very limited (if any) availability of food etc. for sale on the island.
If you stick with your existing travel dates and go to Muck first (including the Mallaig region which could nicely include Fort William), then return to Gourock/Glasgow for the Highland Games, you'll have another week to head up to Inverness to see Culloden, then work your way south along the A9. Note that Blair Castle is right along the A9 and advertises kids' activities. Seeing Ardverikie House, the Highland Folk Museum, and Sterling Castle would not leave you many days in Edinburgh, so you'd want to set your Edinburgh priorities realizing you can't see everything. You could make Stirling an overnight on your way back to Glasgow, after Edinburgh.
Traveling with young kids, realize the importance of preparing for rain. It rains often in Scotland and you can't let that interfere with your enjoyment when you are only in X destination for a certain amount of time. Some advance conversations with the kids and maybe inventing some games about rain might be helpful.
Another note: if you're driving, make sure of navigation to & from Gourock ahead of time. The roundabouts & highway signs along the Clyde are not self-explanatory (IMHO) and it's scary to think you may be on the wrong side of the river since there are only so many crossings.