I think the bigger question isn't which tour size to book, but which company to book. How organized the company is, and how much its guides know (and how skillful they are in making that knowledge fun and exciting!), is the real question.
To that end, I'm partial to Walks of Italy, which is the company started by Jason Spiehler -- the guide recommended by both Rick Steves and the New York Times for several years running. He's picked a group of like-minded, well-educated guides to work with him, and from the rave reviews the company's been getting it shows. Plus, even their normal group tours are no bigger than 12 people.
You can look at all their options on their website, but their Vatican tour is 3.5 hours for just 35 euros/adult, ticket fees included. They also have private tour options, and a neat tour called the "Pristine Sistine," where you go in to the Sistine Chapel before it even opens to the public; that's 85 euros per adult.
Regardless of what company you book with, make sure you ask about the experience/education of the guides and are clear on the pricing. A LOT of companies have hidden fees when it comes to charging for tickets and so on.