Maybe we should drop Toledo?

If you decide to drop Toledo, maybe add a night to Córdoba so you could visit the nearby sights of Castillo Almodovar del Rio and Medina Azahara - Conjunto Arqueológico Madinat al-Zahra.

I admit that we had a car, but why make a transcontinental backtrack from Granada when you could just go from west to east and just backtrack to Madrid?

This is a hard sell to those demanding authentic local experiences, but we had some great 10-hour tourist bus excursions from Madrid (i.e. Toledo area) that eliminated a lot of authentic sweating at local bus stops, uncertain which way the bus might be going. And we got very familiar with Madrid. Do you know a single person who has been to the Academia de Bellas Artes?

I agree with Tim that you should start at the westernmost point. Landing at 7am, you are able to reach Seville that same day.
You could then do Seville-Granada-Cordoba as planned.
And depending on your budget and your plane time, you could decide to skip Madrid altogether and simply arrange transfer from Toledo to the airport.

I’m a proponent of vacations that are relaxing, not laborious.

I just spent three nights in Granada—two full days and two half days. It felt adequate, but minimum.

Is Seville further away than Granada? It seemed kinda similar to me.
Jamie and Tim-I've never heard of any of those places! Very interesting; thanks for sharing!
Scudder-I'm definitely leaning toward relaxing versus adventurous.
Thanks all!

There are many more trains to Seville than to Granada, so it makes sense to start there from Madrid. You'll have more time options, and avoid a very long wait in Madrid!

I’m going to leave you answer your own question, but either way, suggest this routing as your best use of time.

Arrive in Madrid at 7am
Train to Seville 3N
Train to Granada 2N
Train to Cordoba 2N
Train to Toledo (or Madrid and day trip) 2N

To respond directly to your original question: I once did a day trip to Toledo, then regretted not spending a night, so I sort of understand your concern. Toledo is a great, historic city, but so are the others on your itinerary (which looks sufficiently rushed even without Toledo), so with some regrets regarding Toledo, I would have to go with your plan to skip Toledo (this time anyway) and add a night to one of your other destinations.
As a point of fact, Granada is closer to Madrid than Seville (by around 100km), but as mentioned above, the more important criteria to consider are the number of trains, and whether they are high-speed.

I looked at Renfe's schedule. Since there are no tickets yet for Oct 17, I went to Oct 10. There are 2 direct trains after 8 am, at 2.35 and 7.35. Travel time is 3-1/4-3/1/2 hours. You would most likely have ample time to get to the Madrid train station in time for the 2.35 train. There's also a train with a change in Cordoba (all the trains go through Cordoba) that takes longer because of a 25 minute layover.

Soooo, if you go straight to Granada, you land at 7 am, 7.5 hours later get on the train to Granada and get to your hotel around 6.30 pm. Toledo is much less train time. Cordoba is about an hour less train time, but there are frequent trains, so you won't have nearly as long a wait in Madrid. Or Sevilla, with just a tad less train time, but lots of trains. (the trains from Madrid to Cordoba continue to Sevilla)

With or without Toledo, it would seem going straight to Granada is your least desirable option.

Wow, I see there are many more train options for Seville than Granada. I think this time we'll skip Toledo and go straight to Seville.
Thanks for the advice!

An option to explore would involve a busy Day Two but remove the worry of an arrival day connection.

1. Arrive Madrid (1N)
2. Store luggage at Atocha and day trip to Toledo. Late afternoon return to Madrid for luggage and transfer to Cordoba (2N)
3. Mezquita
4. Train to Granada (2N)
5. Alhambra
6. Train to Seville (3N)
7. Seville
8. Seville
9. Train to Madrid (1N)
10. Depart Madrid

Hi Amanda, although I had more time, below is my itinerary. FYI, I DID NOT purchase any train tickets in advance. Once in Madrid I went to El Cortes Ingles on Puerto del Sol to purchase all the tickets I needed (except bus).
Arrive MAD - 3 ngts
Train to Toledo - 1 ngt - Absolutely loved Toledo (thought 2 ngts have been better - so picturesque, although super hilly)
Back to Madrid & then fast train to Cordoba on same day - 3 ngts (feel 2 ngts would have been sufficient)
Fast train to Granada - 3 ngts (feel 2 ngts would be more than sufficient)
Train to Ronda - 2 ngts (would have liked 3 ngts)
Bus to Arcos de la Fronterra - 1 ngt
Bus to Jerez de la Fronterra & then train on the same day to Sevilla - 3 ngts
Fast Train to Madrid & then bus to Airport hotel - 1 ngt
Adios Espanol!