Best tour company to use to tour Colosseum and Vatican/Sistine Chapel

Don, there is no singular 'best'. A scan of the forums will turn up a variety of companies that more versus fewer posters have used. That's in addition to tours offered by the sites themselves,

It can come down to price, size of group, length of tour, and (in some cases) risk of cancellation. That last risk applies mainly to tours which include the underground of the Colosseum: tickets that include that piece are very few and far between - even for tour companies - and some of them have quit including it altogether or have disclaimers in their tour descriptors warning against that possibility or of last-minute changes to different days or time slots.

All of that said, here are some of the independent companies that have been frequently recommended. I will strongly suggest tours of the Colosseum that also include the Palatine and Forum as they're important sites but more difficult to understand without a guide. The Colosseum's own site doesn't offer one of these. Read ALL tour information carefully, including refund policies.

The Roman Guy/Tour Guy: https://theromanguy.com
Walks of Italy/Walks: https://www.walksofitaly.com
Through Eternity: https://www.througheternity.com

Are photos allowed in the Vatican sites? Are bags allowed?

Non-flash photos are pretty much allowed everywhere visited by the public with the exception of the Sistine: no photos at all allowed there. Bag and banned-item rules are on the Vatican's website, linked below; see "Luggage, bags and rucksacks" cloakroom regulations. You don't want to bring anything that has to be checked if you book a tour that includes the basilica as those tours do not return to the museums, where it had to be checked.

https://www.museivaticani.va/content/museivaticani/en/organizza-visita/consigli-utili.html

Through Eternity as Kathy mentioned is my recommendation. I've used them three times and always been happy. I also recommend you look for a tour that includes the Forum and Palatine Hill as well as the Colosseum. I found them far more interesting than the Colosseum.

My family and I used Walks of Italy before and were pretty happy with them. You get smaller groups, which is nice. However, the Colosseum and Vatican/Sistine Chapel are so busy we probably would've been fine doing one of those cheap Viator big group tours.

Also when you go to the Vatican/Sistine Chapel, make sure your guide has headsets for you. When we did the Walks tour, they didn't provide headsets and we could barely hear the guide. After a few minutes, I asked if headsets were available and she said they were available through the museum for 2 euro each, or something miniscule. I ended up buying them for the group because otherwise we would've spent all that money on the tour and not heard a word.

A word of caution about Through Eternity: last June we booked a 9:15 AM tour of the Colosseum, Forum and Palatine Hill. The tour description stated we would go to the Colosseum first, follow by the Forum. This sounded perfect to us as we wanted as early as possible entry to the Colosseum to avoid the heat and minimize crowds. However when we arrived for the tour, our guide said we would be doing a “reverse tour” and would go to the Colosseum last with a timed entry of 11:45 AM. Of course, with the long queue, we did not enter until after 12:30 PM. It was hot, crowded and horrible. We left the tour before it ended, as did several others in our group. I emailed TE to complain and received no response at all. However the guide was great. If you choose this tour and getting to the Colosseum early matters to you confirm the entry time when you book.

..we probably would've been fine doing one of those cheap Viator big
group tours.

A caution about this: Viator doesn't conduct its own tours. They're farmed out to indie companies; some good, some not. When there's a problem, you end up working with the middleman or could get into a finger-pointing argument between Viator and the indie company. Best to book DIRECTLY with the company who is actually conducting the tour.

When we did the Walks tour, they didn't provide headsets and we could
barely hear the guide. After a few minutes, I asked if headsets were
available and she said they were available through the museum for 2
euro each...

This is the first time I've heard of this company not providing 'whisper' devices. In fact, most attractions mandate their use for groups over a certain size; the Vatican requires them for groups of 10 or more. The WOI "Pristine Sistine" notes that the tour includes headsets, as do some of their other Vatican Museum tours. Which tour did you book?

This is the first time I've heard of this company not providing 'whisper' devices. In fact, most attractions mandate their use for groups over a certain size; the Vatican requires them for groups of 10 or more. The WOI "Pristine Sistine" notes that the tour includes headsets, as do some of their other Vatican Museum tours. Which tour did you book?

This was the tour - https://www.walksofitaly.com/vatican-tours/vatican-after-hours-tour/ . In fairness to the company, they ended up reimbursing me for the cost of the tour after I lodged a complaint. I usually never complain but we wasted 20 minutes of our tour on the headset fiasco.

Edit: this was back in summer 2022

Ah, OK. I think they bypassed the headset rule because the entry time was before the general public, and the last tour was at the very end of the day when most (they implied) had left the building. Evidently it was still busy enough to need headsets. Good to know that they reimbursed your purchase.

You can just buy an admission ticket, walk in, and see it for yourself. A guidebook will tell you everything you need to know about it, and you can wander at your own pace.

You can just buy an admission ticket, walk in, and see it for
yourself.

If going that route, for SURE buy a timed-entry ticket in advance from the official websites. You do not want to waste time and energy standing in VERY long ticket lines.

Second the timed entry tickets. Had we not had reserved entry times we would have seen very little of our interests in France this May. Fundamentally scheduled each day hour by hour and did a "guided" tour by ourselves as a group of four. Same in Italy it appears. Even 3 years ago coming out of Covid timed entry was pretty much mandatory for many sites in Rome. We used Through Eternity at the Colosseum and Sistine, small groups of max 12 - which actually at that time was a smaller group...of FOUR!. Very much enjoyed both tours - both early entry.

If going that route, for SURE buy a timed-entry ticket in advance from
the official websites. You do not want to waste time and energy
standing in VERY long ticket lines.

Yes, for major attractions, like the Vatican Museum, I do that. But I don't get a guided tour, I just like to walk around and see it at my own pace.