Toledo is my #1 recommendation around Madrid. I adored it. It has a huge historic district, great for wandering around, as well as a lot of actual sights to be visited if you choose to do so. I enjoyed the circuit of six or seven secondary sights that are part of a wrist-band package sold at the Toledo train (or bus?) station. Most include short guided tours, typically bi-lingual. However, Toledo is on a spur rail line. To get anywhere by train, you must go through Madrid, so it's not a good choice as a base city for side-trips. What I'd recommend is staying in Toledo just for the time you want to devote to Toledo itself--2 or 3 nights (depending on when you plan to arrive and depart; I'd want more than one full day there) and spending the other nights in Madrid for the convenience of side-tripping.
Segovia and Cuenca would be my top choices for day-trips from Madrid. Each is worth an overnight, but you only have so much time. Cuenca (which is on the way to Valencia) is viable for a day-trip only via the AVE train, which gets you there in less than an hour. The AVE fare will be reasonable if you buy the tickets about as soon as they go on sale. Bought at the last minute, the tickets would be prohibitively expensive for a day-trip.