The Basilica and the Museums are two different things. There's quite a walk between them. Each has its own security/entry line. The Basilica is free, and there's no way to go directly from the Basilica to the Museums. In the past, commercial tours covering both venues were allowed (usually) to use a door between the Sistine Chapel/Museums and the Basilica. That avoided the security line at the Basilica and eliminated a lot of walking. It's highly unlikely anyone not on a combo tour would be able to use that door, and it cannot be used in reverse (you can't start in the Basilica and then scoot into the Sistine Chapel/Museums)
During peak season the Vatican Museums are open later one or two nights a week--though the Museums recently extended their regular hours, so the difference isn't as great as it used to be. In the past, a very few people mentioned somewhat less crowding when they took advantage of the late hours. When is your trip?
There is a general pathway through the Museums with a few opportunities to bypass things you're not interested in. With time and determination, I was able to backtrack to see the Pinacoteca, but it was like swimming upstream.
Have no illusions; you'll be sharing the Vatican Museum experience with many, many other people.
From comments posted on this forum, I understand showing up at St. Peter's before it opens (usually at 7 AM, I believe) will give you a relatively short/quick security line to deal with. I think that's by far the best way to see St. Peter's.