Katie,
I'm unsure if you want to take the NYMR for the line or for Whitby, or for the steam interest.
If the latter then there are three other easily accessible options from York- the Keighley and Worth Valley Railway (the 'Railway Children' line), the Embsay and Bolton Abbey (bus up from Skipton, some buses from Bolton Abbey including to Grassington, also to see Bolton Abbey and the Strid) or the Scarborough North Bay Miniature Railway- https://www.snbr.org.uk/
Also, at Leeds, you have the world's oldest continuously working railway at the Middleton Railway- one of those little known but seriously historic railways- https://middletonrailway.org.uk/
(it would make a better 'Jeopardy' question than last week's very oddly worded one, which no one knew!)
Of general interest the NYMR has a big financial problem- the fundraising campaign for the repairs (the 3rd campaign since Covid) has all but stalled £600,000 short of target; also due to operational problems at Goathland all trains are having to be top and tailed increasing costs when revenue is sharply down (engine front and rear).
Thus I would suggest retaining your plans- taking the train to Goathland then Coastliner bus to Whitby (one way) then either returning via Scarborough (half hourly bus then hourly train) or on the Esk Valley Line to Middlesbrough (change for hourly train direct to York- or vice versa.