I can’t speak to using public transportation so don’t know if you could do both Cádiz and Jerez de la Frontera in the same day. To me, Cádiz has no wow sites. It does have a cathedral that is under continual renovation because of the materials used in its construction. Because of its location it is fairly compact, level, and easy to walk around. There are some nice parks and there is a beach. I feel a half day there is sufficient but others will say otherwise.
Jerez is a maze of twisty streets and nice to explore. Its main draws are the equestrian center and sherry bodegas. Its Alcazaba is worth visiting.
In addition to the ferry to Puerto de Santa María, there’s also a ferry to Rota. Both Rota and Puerto have small castles with limited hours. If the bay is rough the ferries stop running.