I have been to and enjoyed all three of the above mentioned suggestions of Avila, Sevilla and El Escorial. I am going to throw in a few other places to research that would be good daytrips from Madrid: la Granja de San Idefonso, Alacala de Henares, or Aranjuez. Look at images and the map and read about all of the suggested places to see what appeals most to your interests. If you are a Quijote fan, about an hour or so outside of Toledo is the small town of Consuegra that I liked a lot. These suggested places offer castles, palaces, gardens, windmills, walled cities, religious sites, or aqueducts. What appeals the most to you or possibly what will be somewhat different from what you will see on your tour? Bus and train transportation in Spain is good and fairly easy to manage. Toledo is a great city, quite the mix of cultures. Hopefully you will get to see it at night and roam its winding alleyways. The cathedral there is stunning.
Let us know what you decide. You can' t go wrong with any of these!