We did a self guided tour of the Coliseum without buying advance
tickets (early October before covid). You can likely find a place to
buy advance tickets online, which would help avoid the ticket line.
You can't buy tickets to the Colosseum AT the Colosseum. They currently ONLY sell them online, and by pre-reserved timed entry. CoopCulture is the official ticket seller, and you purchase by specific day and open time slots.
https://www.coopculture.it/en/tickets/
This is the general-entry ticket (does not include tours or access to the arena floor or underground):
https://www.coopculture.it/en/products/ticket-colosseum-roman-forum-palatine_24h/
That said, I'll throw my vote in with not trying to do both the Vatican Museums/St Peter's and the Colosseum/Forum/Palatine in a day. Note my inclusion of the Forum + Palatine as your Colosseum ticket covers them as well (within a 24 hour period of your Colosseum reservation time) and they're well worth doing. Trying to do ALL of this in a day would definitely be overload,
As mentioned above, you would need to book a tour of the Vatican Museums + basilica that specifically mentions the ability to access the church via the time-saving interior passage, as it's reserved for tours. The Vatican itself doesn't sell one of those but Walks of Italy, The Roman Guy and some other companies recommended by RS posters do.
We also much preferred Galleria Borghese to the Vatican for its crowd control (a limited amount of tickets sold for each 2-hour time slot) and smaller size. Great collection.