You’ve got a Swiss residence permit, so it does not apply, some people... really.
Jim, that is the problem. The Ausländerausweis issued to EU citizens is only valid in Switzerland. It is not a travel document.
They issue a different Schengen-standard card to non-EU citizens. And this card, together with their passports, allows them entry into other Schengen countries without the 90-day limit applying.
EU citizens are just given a paper document valid in Switzerland only, as their passport already exempts them from the 90-day limit.
So for a UK citizen post-Brexit, A French (or German or Swedish) immigration official has no way of telling I am allowed to enter the Schengen Area, he just sees I have been there over 90 days and want to re-enter.