Cuenca

Cuenca became a point of contention and a running joke between me and my Spanish friends when I wanted to visit whilst we were in Teruel in 2011. They maintained it was too far and would have to wait for a future visit. I was disappointed.
So whilst I agree with your desire to visit, I would also maintain it is too far for a day from Madrid (about 170km).
Avila and Segovia are half the distance and can be combined for a long day from Madrid.

Well, this is a tricky issue so I hope I can reply without hurting anyone feelings. Apologies in advance if I do. Indeed, Cuenca -which is in Castille- (as well as Teruel, as per djp_syd comment, which is in Aragon) are remote from most 'tourist areas' so they receive very little visitors, especially foreign visitors. This of course has the advantage of finding yourself immersed in towns and landscapes 'as they were' in the past which is extremely exciting, but on the other hand the area might not necessarily be optimally suited to cater foreign visitors (ie. language issue).

As per gathering info, I don't know whether you have seen already this, but just in case:
http://turismo.cuenca.es/portals/aytoturismo/documents/cuenca_ingles.pdf
http://www.turismocuenca.com/ -only in Spanish
http://www.vercuenca.com/ -only in Spanish
http://www.turismocastillalamancha.es/cuenca/ -only in Spanish

A couple of the sites worth seeing if you decide to visit the area are the Hanging Houses (http://en.wikipedia.org/wiki/Hanging_Houses_of_Cuenca) and the many castles and fortifications still up in the area -maybe not like those in the Hollywood movies but these are the real thing :))

Enjoy.

I also missed Cuenca because it was out of the way. It's definitely a town I'll want to visit during the next trip.

If you go that way, visit Aranjuez on the way from Madrid (A-4 from Madrid to Aranjuez, later cut off on A-40 to Cuenca). It has a palace similar to el Escorial but smaller, most say nicer, with great gardens.