Attractions for large groups in Italy

It is not a special Italy thing, I've worked at the Art Institute of Chicago and there are regulations for tour group sizes related to crowd control and having an enjoyable experience.

Break the group up into 3. I've been to the Vatican Museum in a tour group of about 15 and it sucked because there is not enough space in places for people stopping to listen to a tour guide explain what you are looking at, when it is open to the public. There is no place to sit for those who might have trouble standing or walking.

Also, avoid traveling to Italy in a tour group in summer; it can get too crowded and hot.

In addition, I believe Italy has strict rules about who can lead a tour in places. I know our Rick Steves guide couldn't gather us and talk.

Have you tried a tour company to see if they can organize a custom plan for you? Perhaps Walks of Italy https://www.walksofitaly.com/ or The Roman Guy https://theromanguy.com/.

I believe Italy has strict rules about who can lead a tour in places.

It's true. In Italy there are two kind of touristic jobs to lead group: for both you need a license. The Tour Leader is the person who lead a group and is in charge of the management (timing, ticketing and so on). The Touristic Guide is the person in charge to explain a place, a museum, a monument, an activity.
Do one of this job without a license means risk to be fined if controlled.

find it very difficult to make booking for such a large group because all the tours seems to be capped at 20 or so visitors

Greatly depends by the place you are going. In a museum manage a group bigger than 20 people is really difficult, but there are museums where is possible. At Vatican museum or other very crowded places is better split the group in smaller ones. I did al Tour Leader the Vatican Museums with a group of 20 something people and at the end of the first room was almost impossible understand where our guide was (we had headsets).
If the most important matter is to purchase the discounted tickets for group, you can purchase the tickets for 30 people, but hiring two guides to do the tour.

For a group of 30, hiring a tour guide/company and having them handle the ticketing & logistic would be much easier, and at the end less headaches; pay a bit more and you'll get a better experience. An organization of that size will be broken up into 2-3 separate groups, much easier for the tour guide to handle, easier to keep track of everyone and more importantly people don't get lost or feel like they're not getting much out of the location.

Personally, I would hate to be in a group of 30 individuals moving amoeba-like around, overwhelming an attraction and annoying others.

Hi and welcome to the forum -
As said above, a group of 30 is simply too large for a guide to manage at some of Rome's attractions. The Vatican Museums, for instance, are too crowded to be able to keep everyone together, and some of the spaces/rooms are too narrow/small. It also creates discomfort and annoyance for individuals and smaller groups crowded out by a very large one. And yes, some attractions will have a limit on group size,, such as Galleria Borghese's max. size of 15. Here's how you'd contact them to arranging a tour but no, you wouldn't all be together:

https://galleriaborghese.beniculturali.it/en/visita/gruppi/

The problem you have with Roma Passes is that you can't rock up to just any attraction and get in: some of them require advance, timed-entry reservations to try and manage the mobs. It's not a matter of queuing - which EVERYONE has to do to pass through security checks at attractions which have them - it's a matter of needing to have a very specific day and time booked in advance, and with personal information that is checked against your ID at the entrance to reduce scalping. You don't mention if your group is all adults or a mix of grownups and individuals under the age of 18 but that can add another layer of ticketing complexity. The Vatican Museums are also not covered on Roma Passes.

The Colosseum looks to provide bookings for groups up to 25; the link I’m including is for tickets as of May 1 or later, The tricky part is that it looks like you need to provide your own guide for areas other than the Underground, and as all guides in Italy need to be licensed, your group leader cannot audibly act as one. A group guide will also need to provide “whisper” headsets. You might contact the park via the Speak With Nero/NeroBot at the very bottom right of most pages. There are suggestions for good private guides on some other threads and you might contact a few of them for pricing/availability.

https://ticketing.colosseo.it/en/categorie/gruppi-9-25/

For guides, scan these threads:
https://search.ricksteves.com/?button=&date_range=1y&filter=Travel+Forum&query=Rome+private+guide&utf8=✓

The Vatican Museum offers offer group tours up to 20 participants. A dummy booking for 20 adults prices out at € 830.00, and €630.00 for 10 so a little under 50€ pp which is right about what individuals pay to join a group tour. Click "more info" under the "Guided Tours for groups - Vatican Museums and Sistine Chapel" option in this list:

https://tickets.museivaticani.va/home/fromtag/1/1715835600000/VG-Musei