I don't believe there is a bus serves Chiddingstone Village at all. The National Trust say the #232 bus from Edenbridge does but the National Timetable Database show that service has ceased.
The 231/233 from Tunbridge Wells serve Chiddingstone Causeway which is where Penshurst Railway Station is [and not the same place].
Scroll down on this page to find the local taxi numbers to pre arrange taxis from Penshurst station- https://assets.nationalrail.co.uk/e8xgegruud3g/phrOnwardTravelMap/dc31fae408b283e4d4226b6144bef58f/Penshurst__PHR_.pdf
My thought is to take the bus from Tunbridge Wells to Penshurst Village, pre book one of those taxis to Chiddingstone Village, pre book it back to Penshurst Station, then train (or bus) back to TW (on the train you have to change at Tonbridge, but the train is hourly, compared to the irregular bus schedule.
EDIT- I've just got the map out. There is a good footpath (technically what we call a byway) from opposite Penshurst Church to Chiddingstone Village (about 4km/3 miles) and field paths from Chiddingstone Village to Penshurst Station (about 3km/2 miles). The map is OS Landranger Sheet 188.
For Hever Castle (starting from Tunbridge Wells as opposed to London) I'm not sure that I wouldn't take the same bus to a village called Bough Beech then a 3km/2 miles walk mainly on field paths.