There is indeed a riches of possibilities with those interests! You are in the Scottish forum, so I am going to focus on Scotland. We'll leave England to the other forum. The two best known castles in Scotland are Edinburgh and Stirling Castles. Both are prominent in Scottish history and are impressive places to visit. The buildings themselves are interesting and the locations are amazing. You should check into a possible membership with Historic Scotland if you are visiting several castles that HS properties. Many Scottish castles are still ruins, romantic and interesting ruins, but ruins. Scotland has chosen not to "restore" its castles. They are still great to visit!
One place to go that has lots of castles that are still lived in and not ruins Deeside. Here's the Castle Trail that VisitScotland promotes. I've not done the trail itself, but I've visited several of the castles in the area. I stayed in Ballater which was convenient for visiting them. Some had beautiful gardens. The Drum Castle gardens were lovely as were Crathes Castle's. I also really like the Cawdor Castle Gardens up near Inverness.
Two gardens that I really enjoyed visiting were The Explorer's Garden and the Inverewe Garden. They are both wonderful gardens that demonstrate the role that Scotland and the Scots played in developing all those British Gardens!
Of course, there is the Royal Botanic Garden in Edinburgh and the Glasgow Botanic Garden.
The Inverewe Gardens are the farthest away. But It will take you up to one of the most beautiful parts of Scotland. You can drive there and back from Inverness in a day. It would be a good day trip.
Pam