We went the next morning to visit the Tower.
Just an FYI, we arrived early for our ceremony tour, about one hour. There was a group already there at the gate waiting. We just figured they were early like us. Anyway, we decided to walk around a little cause standing still is harder for me. When we returned a half hour or so later, the group was gone. I panicked for a second thinking we missed it. There was another couple nearby so I asked them if they were here for the keys tour. Yes they were. They told us the group before was a private tour of the tower, family and friends of the Beefeaters. Whew. The other group meet up with us later on.