We only have a few days tacked onto a work trip to Belfast in late July. We love history (castles are a favorite) and folklore. Would it be more fun to spend our time in Scotland or Ireland? We will have access to a car.
When time is limited, it is best to minimize travel time. Therefore, I would be inclined to spend my time in Northern Island or Ireland. You can have a wonderful time anywhere in the British Isles. I agree that Wales is tops for castles.
Scotland.....more castles than you can count.
Plenty of flights from Belfast to Scotland.
If you don't want to drive, take a look at Rabbies one day tours.
Scotland's Castle Trail fly into Aberdeen.
Consider a visit to Stirling Castle - a wonderful experience and not overrun like in Edinburgh. We stayed at Castlecroft B&B. It wasn't that far to walk to the castle, though it was uphill. There's lots to do in the area.