We only have a few days tacked onto a work trip to Belfast in late July. We love history (castles are a favorite) and folklore. Would it be more fun to spend our time in Scotland or Ireland? We will have access to a car.
When time is limited, it is best to minimize travel time. Therefore, I would be inclined to spend my time in Northern Island or Ireland. You can have a wonderful time anywhere in the British Isles. I agree that Wales is tops for castles.
Of course depending on how long you have, Edinburgh might be a good one stop. Edinburgh Castle and Holyrood House (palace/castle - read up on Mary Queen of Scots!) are walkable from city center and each other. Stirling with its castle is an easy day trip by train. Lots of large and small museums on and near the Royal Mile.
With a car, easy to drive into Northumbria (England) for Alnwick Castle and Bamburgh Castle if not "castled out". Also some Scottish castles and "great houses". Blair Castle is close to Edinburgh. If you have time and wanted the Highlands experience - Dunrobin Castle. The "great houses" may or may not fit your definition of "castle"....
Edinburgh also has an airport with non-stops to some US airports.
Scotland.....more castles than you can count.
Plenty of flights from Belfast to Scotland.
If you don't want to drive, take a look at Rabbies one day tours.
Scotland's Castle Trail fly into Aberdeen.