The Cordoba Alcazar building itself isn't particularly memorable, at least compared to the one in Sevilla or even the castles in Malaga, Jaen and Almeria. I went to Jerez for the first time a few weeks ago and the buildings of its alcazar are also more interesting than Cordoba's.
However, the gardens at Cordoba are fantastic, perhaps amongst the best in southern Spain. I suppose that does depend on (a) time of year and (b) whether you're interested in gardens. Personally, I'd rather spend a couple of hours in the gardens than I would in most art galleries. But others, I know, prefer looking at pictures.
In any case, it's cheap to visit and not out of the way from other sights (just a few minutes walk from the Mezquita-Catedral), so unless you have a very tight "time budget", I think it's well worth seeing and you can then make up your own mind.