Castles on trainline from Edinburgh

Hmm, a coastal castle. I would have said Scone Palace in between Stirlng and Leuchars, but it's not on the coast.

Earlshall Castle is right by Leuchars -- have you thought of doing that? If you Google "castles near Dundee" you may get more ideas.

whilst in Edinburgh you could visit Craigmillar castle.
https://www.historicenvironment.scot/visit-a-place/places/craigmillar-castle/
easy to get to from the city centre by bus (about 20-30 minutes) and a short walk. Take any bus that goes to the New Royal Infirmary (7,8,21,24,33,49 ,might be some others but these are these I remember).
taxi from the city centre will be about £12.

While in Stirling, be sure to visit the William Wallace Monument as well. It's a short bus or taxi ride from the castle.

As for other castles in the area, have you considered taking an escorted day tour? It appears there is a pretty good selection of small-group tours available from Edinburgh - whatever your interests are.

Take a bus to Craigmiller castle just outside Edinburgh if you want a castle. Roslyn chapel was nice to see and is also a bus ride from Edinburgh.

Since you mentioned Hairy Coos, let me suggest the hairycoos.com tour. It's FREE! We just did it and not only will you see hairy coos, you'll visit Duone Castle (of Monty Python and Outlander fame). It's not been refurbished, it's authentic and just as it was centuries ago.

The tour was very informational, had plenty of time at each stop, lots of options for snacks, drinks, etc. without any pressure to buy anything - in fact he avoided places that give kickbacks to tour companies.

At the end, you pay what you can and what you think the trip was worth. Again, no pressure. The point of the tours is to show people parts of Scotland, especially the Highlands, that many don't see.

Any suggestions of other coastal castles within easy striking distance of Edinburgh?

The best castles in Scotland are in England.

Alnwick; Bamburgh; Warkworth; Lindisfarne; Durham...