I can't really help with the Avignon to MSM dilemma (but I agree with those who recommend avoiding changing trains in Paris if possible. It looks like there is a Flixbus that runs from Paris to MSM, but I would imagine you'd still have to make an annoying transfer from train station to wherever the bus leaves from). If you want an early arrival at MSM, I'd definitely look into a nice and convenient place to spend the night, but you might also consider a later arrival and spending the night.
I did MSM and Bayeux (and Rouen) several summers ago. I took the direct MSM bus (it's a comfortable bus) from Rennes. This is easier than going to Pontorson, because the bus takes you right to the causeway of the Mont, without adding an unnecessary extra step. I doubt there's a train trip from Avignon to Pontorson that doesn't require at least two changes.
I stayed the night on the Mont because it was July and my timing was such that staying overnight was the best way to avoid the mob and really enjoy myself. It's worth considering, though I agree with the other posters that the lodging and dining is nothing special. BUT, if the abbey is open for its late night, summer hours in mid-May (not sure when they start), I think that's a good argument for a late stop.
After spending the night, I then took an early morning bus to Pontorson (early to miss the tourist buses) and then the train to Bayeux. Bayeux is wonderful and I wish I'd spent more than one day/one night there. There's not only the tapestry, but a beautiful cathedral with gorgeous stained glass, a quality fine arts museum, and access to the D-Day beaches if that's on your list.
Or you could make Bayeux your base and do a day trip to the Mont. There is a direct local train (to Pontorson, you'll have to get another bus or taxi from there), but it takes about 2.5 hours.
Back when I was better about my travel blog, I wrote two posts about this trip--one about MSM and the other about visiting the Bayeux tapestry. At the risk of shameless self-promotion, I'll post them below.
https://www.historycoloredglasses.com/2017/04/visiting-mont-saint-michel-by-night/
https://www.historycoloredglasses.com/2017/03/art-as-propaganda-and-history-the-bayeux-tapestry/
I hope some of this is helpful and that you make it to Bayeux. I've been lucky enough to travel in many (though hardly all) parts of France, and I think Bayeux may be my ultimate favorite.
Caroline