Base in SW England

South but not nearly so far west, you might consider Chichester, where there are some "Gardens, Castles, Manor Homes, Historical Ruins" in the town or within easy reach by bus or train.

https://community.ricksteves.com/travel-forum/trip-reports/chichester-a-week-off-the-well-trodden-path

We spent a week in Chichester, visiting:

Gardens:
Bishop Gardens (in town)
West Dean Gardens (bus)

Castles: Arundel (train)

Manor Homes: Goodwood House (Uber was easiest), enjoy tea and a tour

Historical Ruins:
Fishbourne Roman Palace (bus)
Novium Museum (tiny, in town)

ETA: we made a day out (by bus) from Chichester to Fishbourne Roman Palace to Bosham (Lunch at the White Swan pub) and then Emsworth, where we walked the little sea wall and had coffee and cake on the pretty patio at Cross Kitchen in Emsworth. On the way back, I visited the lovely West Dean Gardens.

Plymouth wouldn’t be my choice.

How far West do you want? I would think leaving England for Cornwall would tick the boxes. Perhaps Falmouth, Penzance or St Ives which all have bus and train connections.

Some of the Cornish hedges (and there are 30,000 miles of them) are the oldest man made structures still serving the same purpose for which they were built, (as in the oldest on Earth). There are also many Neolithic sites and the collection of gardens is without equal in the UK.

In the South West Truro or Falmouth would suit you-
There are a whole clutch of gardens in the area; For Castles in that area you have St Mawes Castle (bus from Truro, ferry from Falmouth), Pendennis Castle (Falmouth), Restormel (at Lostwithiel- a train station town), Launceston (a bit further afield- train to Plymouth, then bus or there are other routes by bus from Truro) and St Michael's Mount (Cornwall's Mont St Michel).

Stones- see this list- https://www.cornwalls.co.uk/history/ancient/stones.htm - a lot of them are on the Penwith Peninsula (bus from Penzance).

Historical Ruins- Tintagel Castle is the obvious (bus from Truro via Wadebridge), a lot of remains of the old tin mines, Rame Head Chapel (bus from Liskeard or Plymouth, then SW Coastal Path), St Helen's Oratory (Cape Cornwall), among others.

Manor Houses- that could again include St Michael's Mount (depending on definitions), Lanhydrock (walk down the old carriage drive from Bodmin Parkway station) and Godolphin House again among others
(From Plymouth also Cotehele House-walk from Calstock station or boat trips up river from Plymouth)

There is an awful lot more than 4 days there- the list needs seriously whittling down

Thank you everyone for the thoughtful replies! There were such great insights - I'll be focusing on Penzance as a base and starting to research that as it seems reasonable for both incoming and outgoing needs, and the Coaster bus looks outstanding! Chichester sounds great and will keep it in mind for future trips when I stay closer to London! Looking forward to brainstorming with all of you on this and the other legs of the trip!