Gibraltar is an interesting little colony and the various military sights are a nice contrast to towns back in Spain. It is certainly worth a day trip if in the area. However, I would not make a long journey just to visit, especially as you already have a a rather full itinerary for two weeks. If you stick to Tarifa/Tangier then you pass Gib. on the way too/from either Ronda or Granada so you could call in for a day. Supposedly the border controls are going, but I'd still park in Spain (La Linea), and walk across rather than trying to park on Gibraltar itself.
If you drop Morocco, then I'd also skip Tarifa. There is nothing wrong with the town, but nor is it especially interesting other than as a place to get a ferry. I think there is a castle but loads of places have castles.
If you aren't interested in horses, then Cadiz is a better choice than Jerez. Indeed, Cadiz is better anyway, with a more atmospheric setting and some diverting Roman and more recent sights.