Early Thursday morning is the time to beat crowds, if just the Basilica/dome is your intent. If you do St. Peter’s only, it’s closed Wednesday mornings during the Papal audience. It’s a busy morning in front of St. Peter’s, and the Basilica doesn’t open until afternoon.
Mid to late afternoon could be less busy either Wednesday or Thursday at the Vatican Museums, if that’s what the sold-out tickets are that you’re referring to. We were there just before Christmas in 2012, and again this past September. Getting tickets was an issue last fall, too. The very moment that tickets were supposed to be available to reserve Online for our date, I was logged on, and they were suddenly all sold out. I seriously believe that Viatour and other commercial mass touring companies have super-fast computers that buy out almost all ticket slots before the public even has a chance. The Museums did have a handful of dates with available time slots in the afternoon, and even slots with guided tours by museum guides, albeit in German, Italian, or Spanish - the English tours were completely unavailable. But ultimately we hired a private tour guide, who gave us a Website that had tickets, only for religious-oriented guided customers. That cost more, but we got in. A guided (private or group) tour could still be your opportunity, if you book one.
One big advantage of a guided museum visit is being allowed to exit the Sistine Chapel through the “guided visitors only” door at the back, which is a short cut to St. Peter’s. That lets you go to the Museums in the morning, have a tour, then get to St. Peter’s without having to do another security check. Back in 2012 on our own, we were able to get through the “secret” door, with advice from the Rick Steves guidebook, and it saved us lots of extra walking and time. We were able to see St. Peter’s at length, without getting in the long Security screening line (we’d done a very short screening upon entering the Museums), and start the dome climb just before last entry in the afternoon. It made logistical sense to see the Museums first thing when the opened, make our way through the Museums to the Basilica, see it, then finish with the dome (using the elevator for a boost).
This fall, the private tour got us thru the back door at Sistine Chapel, and into St. Peter’s before lunchtime, but we didn’t stick around for the dome.
Because so many people attend the Papal audience on Wednesday mornings, perhaps there’s less of a crowd for the Museums (if you can get in) on Wednesday morning, but if the Papal attendees stick around, the afternoon could be busier at St. Peter’s, once it’s opened.