Just seconding Donna and Nigel's information. Donna has provided the correct site for booking tours through the Vatican itself, and while I haven't done them myself neither have I read many complaints so you'll be fine. But as they've noted, a tour isn't necessary for either the museums or basilica as long as you have a good guidebook and are willing to do some reading up in advance. The advantage here - besides obviously saving money - is being able to choose what you wish to see, and spend as long, or as short, a time doing that. The biggest advantage to a tour which include both museums and basilica is that groups are allowed to access the church directly via some backstairs, avoiding the long walk between and the security lines at the basilica.
No, I've never seen St Peter's without some sort of line - longer or shorter - to go through security but shorter might be more possible in January than shoulder seasons, when we've been there.
Regarding headsets, there are definite advantages to tours which use them for the reasons already presented. You're going to hear your guide much better, and especially so if you get stuck beside some very noisy tourists. They also allow guides to speak at normal levels to reduce the overall din. If you've even been in a museum overrun by headset-less tour groups, you'd opt for the sets! It's a very good thing that the Vatican requires them for groups over 10 people.