I went to the Vatican Museums earlier this year (March). I don't remember my exact entry time--sometime around 1 PM, I think. (Early entries tend to sell out first.) There were two lines--one for folks like me, who had booked online and had entry vouchers, and a longer line for folks wanting to buy tickets. I don't remember whether there was someone looking at the vouchers before I got in line to be sure I had a voucher for the current date and for an appropriate time. My guess is that they wouldn't care if you were a bit early. My line moved reasonably fast. There was a minder who allowed multiple people into the building periodically from my line and/or from the other line. The line for the ticketless folks moved more slowly, but it did move.
I entered the building and found myself in a sort of lobby with a lot of folks milling about. I had to go to one of the theatre-style ticket windows to convert my voucher into a ticket. I'm not sure, but I think the folks needing to buy tickets used the same windows. It seemed you could get in line at any window. Unfortunately, they don't use a speedy line, so you could end up in an especially slow-moving line and waste some time. This process could take up to 15 minutes or so, I'd guess. If things get backed up, I assume the line-minders admit people to the building more slowly.
Once you have a ticket, you proceed to a turnstile; the ticket opens the turnstile.
I walked back through practically the entire museum to find the Pinacoteca after following what I think was the suggested route without shortcuts to the Sistine Chapel. I was definitely walking against a fairly heavy flow of visitors, but it wasn't a significant problem. However, by that time it was fairly late in the afternoon (maybe 4 PM or so), and I think there were more people exiting the museum than entering it. I imagine retracing your steps would be more challenging at 11 AM or 2 PM (just a guess).
I don't recall being rushed out of any room in the complex. I left the Sistine Chapel quickly because it was mobbed, I'm not religious, and I don't care for cast-of-thousands ceiling frescoes. Therefore, I don't know whether they do anything to push individual visitors out of the Chapel after X minutes.
Some rooms will be blocked off, and there will be places where you are forced to follow one specific route, but you should be able to return to any room that's open. (Whether some are open early in the day and then close, I have no idea.) You will be given a diagram of the museum when you pick up your ticket. I didn't find it very helpful--perhaps because the enforced routing was not shown--but it was better than nothing. There's a good bit of signage pointing the way to the Sistine Chapel, sometimes indicating a shorter path there. You'll be on your own to figure out how to retrace your steps if you don't want to take the usual direct path to the exit.