If you stay in the old town, you can just cross the Charles Bridge to get over to the Castle and Cathedral complex. I would explore the old town a little bit Saturday afternoon/evening, then get an early start Sunday morning. Walk over the Charles Bridge and admire it. There is Sunday mass at 8:30 a.m. at St. Vitus Cathedral. They won't let you in just to take photos and look around, so you'll need to stay for mass (just tell the guards at the front that you are going to mass). But if you get there at little before 8:30, you'll have a bit of time to explore quietly, then listen to the mass and still look around (from your seat). I'm not religious, but I enjoyed this and it was nice to see the inside without a throng of tourists. Then after the mass ends, explore the castle complex. I would highly recommend the Lobkowicz museum, which is basically in the castle complex. That is a really full, packed morning. Grab lunch nearby (we enjoyed the Klasterni Pivovar), and you'll still have more time to enjoy the castle area or whatever else you want to do. Then with your remaining days, you'll have more than ample time to explore the rest of Prague and hit the remainder of your must-do list.