If you chose to visit Sevilla, then the Alcazar, cathedral and Giralda tower would probably be the only things you’d have time to see. Sevilla is about a 1.5 hours drive from Cadiz. To me, Cadiz is worth a few hours at most with nothing exceptional to see. It’s cathedral is nice but doesn’t compare the Sevilla’s. That leaves simply walking around the town. It’s on a peninsula so it’s longer than it is wide. If you stayed in Cadiz and ran out of things to do, there are ferries that run from the port area over to Rota and Puerto de Santa Maria. They run regularly but will stop running if the bay gets to rough. In those instances a bus would drive you to back to Cadiz. I like Rota and it’s a nice place to walk around (disclaimer: I lived there for over 3 years). The main sights are castle Luna (small), the lighthouse, and the promenade along the beach. It’s popular with the tourists between May and mid-September, but still maintains a small town charm. The ferry drops you off down by the castle and lighthouse.
My last visit to Puerto de Santa Maria was in 2011. At that time, Spain’s economy was depressed and it showed. There was litter and trash seemingly everywhere and unemployed just hanging out. The town itself just looked tired and run down. The main sights are the bull ring, cathedral, and the castle of San Marcos (only open for a couple hours, two days a week). Columbus was given the money for one of his voyages there from Ferdinand and Isabel. It is a nice castle. The ferry drops you off near the cathedral. No idea what it’s like now.
Jerez has an alcazar worth visiting and sherry bodegas if you wanted to tour one or two. The ones I visited were Osborne and William and Humbert and they were right in town. There are others (Fundador, Tio Pepe, etc.), but you’d need to check tour hours and location. Jerez is spread out more than Cadiz.