Toledo is really close to Madrid and brain-dead easy to reach by train. It makes a perfect day trip. And guess what? Everyone who goes to Madrid (which is pretty much all tourists visiting Spain), they all get on that train, and they go to Toledo as a day trip. Because it's so easy, they're already in Madrid, and they don't want to bother with a separate hotel for Toledo since it's so efficiently done by train.
Can you guess what happens?
Every day, late in the morning, the train rolls up and the crowds of day-trippers step off. Every organized tour in Spain also rolls a tour bus there. They all show up around 9:30 or 10 am, too. By 11 am the streets are jam-packed with a perfect example of over-tourism, a once-charming place that's now overcrowded and being loved to death. You do not have to do it this way.
Decades ago, when I first started learning about European travel from Rick Steves, one of the very first things I learned was this: If a place is super-popular as a day-trip destination, DONT GO THERE AS A DAY TRIP! Instead, be there when the day-trip crowds aren't – I can still quote Rick's book from memory (maybe paraphrased a bit) – "Go when all the day-trippers have all gone back to their big city hotels in Madrid with reliable plumbing" – spend the night there, and do Toledo when the day-trip crowds are gone.
The tide of humanity is quite predictable: they roll in late in the morning, they clog the streets until mid-afternoon, by 4 pm, they're all melting away, by 6 they're gone, leaving all of Toledo's magical charms intact for you, wise visitor who decided to spend the night. After they're gone, the streets are empty, quiet, charming, magical. And yours. This is the Europe you have dreamed of. Don't bypass it so you can get back to your hotel in Madrid!
Be smart: don't do exactly what everyone else is doing. Arrive mid-to late-afternoon as the tour buses and trains and all the day-trippers are rumbling away, watch and listen as the calm and quiet descend on the city. Spend the night. Savor this amazing intact medieval city for a few hours in the evening, when it's all yours and yours (nearly) alone. Sleep in the quiet old center, and get up early - as early as you can. Get out early, wander the deserted streets for a few hours in the morning. Around 9 or 10 am, you'll hear a low rumble as the tour buses approach. They will park, the doors will open and the crowds will pour into the streets you have had to yourself since the previous afternoon. Watch as the trains pull up and pour out their daytrippers, too. Check out of your hotel, move on, and don't look back - you've seen and experienced Toldeo at its best. Don't stick around to see it at its worst, jam-packed with daytrippers.
It worked for me. I loved the place.