Milan day trip to Verenna

Train tickets are all Regionali. No advance tickets needed, no matter how many people are on the train. Ferry tickets don't sell out either, although there certainly is a limit to the people they hold.
If you return to Milan from the Nord Lago station by the Como water front, you will arrive at a different train station than you left from. Trains to Varenna go from Milano Centrale and the trains from Como Nord Lago arrive at Milano Cadorna station. To return to Milano Centrale, you need to go to Como San Giovanni station on the side of the hill on the west side of town. It is walkable and there are buses. Milano Centrale and Cadorna are connected by the Milan Metro system.

To me a day trip with what you have planned looks like a lot. We stayed in Varenna for 5 days 3 years ago. Our day trip to Belaggio took most of the day leaving from Varenna. It was May. The lines in Varenna were long and they kept changing the ferry entrance. Large tour buses drop off travlers in Varenna. We spent most of the day to see Belaggio and then the lines back to Varenna were even longer and more confusing due again to multiple line changes and many of us trying to understand the directions they kept explaining. Looking at schedules, it does look seamless, but beware. It was glorious however.

I was in Varenna on a day trip from Milan two weeks ago today (!). It was a rainy-sprinkly day so the crowds may have been less than normal for mid-May (I did not find it very crowded at all). You don't need to book ahead on the train as it's a regional train for Como town or Varenna. Worst case you end up standing on the train. I think trying to take the boat the whole way from Como to Varenna might be a bit much, but initially I had the same idea. The boat to Bellagio is much shorter, maybe fifteen minutes. I arrived in Varenna at 11:30 and left on the 7:30 train after an awesome visit. I did go to Bellagio on the ferry for a couple of hours and it was nice. Had a very good visit in that time. I found Varenna much more captivating. I would encourage you to rein in your ambition and slow down, don't try and accomplish too much.

Rather than trying to do it as a day trip, you might want to consider a one or two night stay. It is a lovely area that deserves to be savored slowly.

Just to follow up- I ended up doing a simple day trip. I caught a train from Milan Centrale in the morning. Got off at Varenna. Walked the small town and had a quick snack. Then boarded a ferry for Balaggio. Spend a good part of the day there wandering up and down the walkways. Had lunch over looking the lake. Then took the slow ferry to Como. It was relaxing way to see the lake and all the interesting towns along the shore. Arrived in Como and walked to the train station that took me back to the Milan Codrona station. Easy peasy and relaxing. It was a nice introduction to Lake Como area. All tickets were purchased day of travel.