Best Nutshell time options if staying the night in Flam

Don't expect either one to be less crowded. The afternoon ferry out of Flam will have all those doing NiN in one day. If anything, I would guess the afternoon ferry would be more crowded.

The train from Oslo to Myrdahl continues on to Bergen. It's a regular train between Oslo and Bergen.

When I did it, I took the morning train out of Oslo, stayed the night in Flam, and then took the morning ferry out of Flam.

Thank you! I will do the same then, and I just realized that approach will give us more time in Flam.

I booked my own NIN from Bergen to Oslo with an overnight in Flam. The only thing that was for sure less crowded was taking the earliest train up from Flam to Myrdal - but that was I think mostly because there were no cruise ships in port that day (which I'd advise you to check if you flexibility in dates). You don't say when you're going, but if it's peak summer season everything will likely be busy so it's good to book ahead.

The fare for the Oslo-Myrdal part of the train trip varies and can increase between now and your travel date. Some trains sell out during peak periods. I think the Flamsbana down to Flam might sometimes do so as well.

It's not usually a problem to make your own bookings and save some money (I think it's at least $100 per person), assuming you don't have to depend on FjordTours to find you a hotel room. (That part can be quite the challenge.)

I think your idea of morning departures makes sense, but I'd recommend doing some research on what's available for you to do in Flam unless you plan to hike. Without a car, the options may be somewhat limited, though I know a lot of folks like renting bikes, taking them up to Myrdal and riding down to Flam. (That means making two trips on the Flamsbana, and Flamsbana tickets aren't exactly cheap.)

Thank you all! I did manage to find a hotel room in Flam at the one Rick Steves recommends (the Heimly Pensonjat) - rather nosebleed prices but it does include breakfast and dinner for 2.

Par for the course in Norway. You have to just not think about what you're spending. In the end, what you'll remember is the fantabulous scenery.