May 1st Labor Day could contain some surprises in terms of unexpected schedule changes and since you're not even planning on being in Paris until May 4th, May 1st would probably be a good day to exclude from your time in Paris. You're early in your planning so excluding this day should presumably not be a problem for your trip.
Re the other 2 days, we were in Paris this year on during the first 2 weeks in May and, following our habit, we carefully avoided Labor Day but were there on both Ascension Day and VE Day and those holidays did not cause any problems with things we wanted to do. They might be bank holidays, I don't know, but ATM's were working.
I notice your current schedule doesn't even include May 1st. Your 4 may - 11 May should be fine.