I agree with acraven (as usual) - Avila is all about the wall, it's a fun stop and worth playing around up on it for a while, but other than that, we didn't find the place particularly special. Definitely would allocate more time for Segovia, since there's much more to see there (not sure it warrants 4 days, though). We did Avila as a stop on the way from Toledo to Segovia, and it worked perfectly as a mid-day leg-stretcher (good workout huffing up and down all the stairs on the wall). We had a rented car (I do not know if the train is an easy/good option for Avila - it might be, but we had a car).
But since you have plenty of days here (in/near Madrid) I gotta ask...hopefully, you already are including an overnight stop in Toledo...and specifically spending an overnight there, rather than doing a day-trip to Toledo amongst the crowds? I ask because an overnight in Toledo is so much better than a mid-day day-trip there (a mistake which almost every tourist visiting Spain makes). You have enough "days" in/around Madrid to do Toledo as an overnight.
You can (and probably should) take the fast, efficient trains between Madrid and Segovia, and (if you go) between Madrid and Toledo. If Toledo is not already in your plans, consider this: take the train from Madrid to Toledo, arrive in the afternoon. Note the discouraging crowds. Then note how those same crowds melt away late in the afternoon, as they all pile back into their bus or train returning to Madrid for the night. Enjoy magical Toledo at its best, free of crowds, late afternoon, into the evening, and sleep there. Get up early the next morning, savor the quiet, lovely old city center sharing it with just the locals in the morning hours. Then as the tour buses roll in, and the trains show up, and all the day-trippers fill the streets, jump in your rented car by 10 am, and head out.
We drove from Toledo to Avila, spent a couple hours playing on the wall, then drove on to Segovia, dropped our car there, and enjoyed the city for a couple nights (like Toledo, it gets crowded during mid day, for the same reason: it's so easy to access from Madrid that most tourists do that, with predictable results). When you're done with Segovia, take the train back to Madrid.
Both Toledo and Segovia are being "loved to death" (like many other magical places in Europe). They are at their best if you spend your time there when the day-tripper crowds have all gone home. That just requires inverting the schedule a bit, and spending the night there. Both cities are lovely when the crowds predictably disappear.
Hope that helps.