From a scheduling standpoint it is often recommended that the departure city be left until the end of the trip so that you don't waste a substantial part of the last day rushing back. Your case is an ideal one for that plan, because you can easily head to Toledo by train (or even by bus in a pinch) on your day of arrival. It appears that the fare will not increase if you buy at the last minute, so you don't need to guess what would be a viable departure time from Madrid. I think you may save a substantial amount of money by buying round-trip tickets to Toledo rather than two one-way tickets. A further advantage of this plan is that it's a way to utilize part of your (probably jetlagged) first day on a necessary relocation,
Your side-trip to Segovia should be made from Madrid rather than from Toledo, for reasons of geography.
Pamplona gets booked up for the festival. Look for lodgings now. Be careful about cancellation policies. You may encounter minimum-stay requirements (pure speculation on my part). If you end up taking the train, I imagine you need to buy the tickets well in advance, too (still guessing).
All the possible side-trips from Barcelona mentioned up-thread can be taken on the spur of the moment (just check bus/train schedules in advance), so you can wait to see how much you want to do in Barcelona before making a decision. The same is true of the Madrid-to-Segovia trip. As I mentioned previously, Cuenca is a non-starter if you book last-minute AVE tickets, because of their cost.
Check ViaMichelin for driving times and estimates of fuel/toll costs so you can do a proper comparison to train fares. I would anticipate quite substantial parking costs in Madrid and Barcelona; I'm not sure about the other cities.
With two weeks available to you and two large cities plus Pamplona already planned, I don't see you doing a lot of wandering around through small towns, and the train will often get you from place to place faster than a car. Fares for four people for your Madrid-Pamplona-Barcelona travel legs could be either a bargain or substantial, depending on when you buy tickets.
Check rail schedules and prices on the Renfe website. Stations for the three large cities are:
Madrid-Puerta de Atocha
Pamplona/Iruna
Barcelona-Sants