It is a pain in the neck to get to Tarifa. I've been to Andalucia 3 times and nixed the Tangier idea from the beginning. It was described to me like this- Tangiers is to Morocco as Tijuana is to Mexico. Malaga is a pretty town, but definitely a second-tier destination when compared to David's suggestion. And Gibraltar was described as a poor imitation of the UK so I haven't been there either.
Cadiz > Jerez > Arcos de la Frontera > Ronda > Granada can all be done by public transportation. Base in Jerez and day trip to Cadiz by train and to Arcos by bus. See a bit of Jerez (take a fun sherry tour at Tio Pepe, see a terrific horse show at the Royal Equestrian School).
if you land in Madrid in the morning or early afternoon, you can go to Sevilla by train and start there. [better would be to spend 1-2 nights in Cordoba first, but you can visit as a day trip from Sevilla]. That eliminates splitting up your time in Madrid. How long you spend in Madrid depends on how much time you want to spend in the art museums. If that is a priority, then you'll want 2 full days just for that. If not, you may only want 2 days there. Go to Toledo between Granada and Madrid. If you have a lot of luggage, store it at the train station in Madrid (the only trains to Toledo are from Madrid) and take a small overnight bag. Toledo is very walkable, but parts are pretty steep, so less luggage is easier.