Neither has any “must see” sights like some other places. To me Cádiz has a small city feel to it. It has a nice park to walk through as well as a beaches. Its cathedral is under continual restoration due to the materials used in its construction. From it you can take a ferry across the bay (weather dependent) to either Rota or Puerto de Santa María. Puerto has a nice, but small castle if you can get there when it is open and its catedral is nice. The castle is where Ferdinand and Isabella gave Columbus money for one of his voyages. Depending on what all you wanted to do while there, Cádiz may be out of the way, e.g., the white villages.
Jerez de la Frontera still maintains a large town feel to it. It has a small Alcázar worth about an hour of your time. There are numerous sherry bodegas is you want to take a tour. Arcos de la Frontera, the closest white village, is only 30 minutes away. Puerto de Santa María is about 15 minutes away. If you wanted, you could get over to Doñana National Park bird observatory in about 30 minutes. Jerez is inland and doesn’t have the beaches like Cádiz. From Jerez you could make full day trip to a number of the white villages.