OK, so you do have 3 full days...which I consider bare minimum for Rome but you can cover some ground in that time. Keep in mind that herding 11 people around is also going to be interesting and probably not especially speedy, especially as none of you have been to Rome before.
Figure we will use tours to avoid general lines, but haven't actually
made the reservations yet.
You do not have to use tours to avoid lines: that's easily done simply by pre-booking entry tickets. This may also be a better idea with a group as large as yours as not everyone may be up to moving along at the same pace. I won't say "don't" but you may have issues even finding organized tours that can accommodate that many at this late date (I assume your trip is next month?)
Vatican: Friday night is a very good idea, and you could visit the basilica BEFORE you go to the museums. You can choose a tour of the museums or just opt to self-tour them. Tickets here:
Self-guided tour:
http://www.museivaticani.va/content/museivaticani/en/visita-i-musei/scegli-la-visita/musei-e-collezioni/musei-vaticani-e-cappella-sistina/-i-aperture-notturne--i---visita-libera-musei-e-cappella-sistina.html
Guided tour:
http://www.museivaticani.va/content/museivaticani/en/visita-i-musei/scegli-la-visita/musei-e-collezioni/musei-vaticani-e-cappella-sistina/-i-aperture-notturne--i---visita-guidata-musei-e-cappella-sistin.html
Colosseum:
Pre-purchase general-entry tickets here:
http://www.coopculture.it/en/colosseo-e-shop.cfm
Buy only tickets for the adults, and choose the option of picking them up at the reservation desk. Your teens under age 18 will be free BUT need to show proof of age at the reservation desk (bring their passports) to pick up their free tickets so you'll need to go to that desk anyway. Audioguides are available for rent. Tickets are good for the Forum and Palatine as well: those two must be visited together, at the same time.
Tours of the Colosseum in English are offered for € 5.00 but must be reserved by phone, and may be sold out for your dates but you could try. Info under "Individuals guided tours" about 3/4 of the way down this page.
http://www.coopculture.it/en/colosseo-e-shop.cfm
Borghese: you need to buy these right away if trying to purchase for 11 people. This is a very popular attraction and tickets can sell out well in advance during high season. Buy them here:
http://www.tosc.it/tickets.htm?affiliate=T2C&sort_by=event_datum&sort_direction=asc&fun=erdetail&doc=erdetaila&erid=912678&language=en
More info:
http://www.ticketone.it/obj/media/IT-eventim/pdf/en/galleria-borghese-5-en.pdf
Again, your children under age 18 will be free but choose the € 2.00 reservation fee option for them and bring their passports.
Whether you break for a nap or push through the day is entirely up to you. Once we leave our hotel in the morning, we're usually gone until after dinner at night so you'll just have to judge how well everyone is doing. You may want to consider breaking your group up so people who want to go back for awhile can and those who don't can keep going. I'm thinking you're also going to need to make dinner reservations if all 11 people will be dining together.
Where did you book your accommodations?