Yes, you should really try to see the inside of a couple of colleges. Most of them have rather impressive buildings. I'm not sure which tours are the best though, or how fast the (rather long) queues move. (I've been fortunate enough to have been able to see them without taking tours.)
Regarding museums, the Ashmolean is great but I can also recommend the Natural History Museum. And if you like reading, a visit to Blackwell's for some shopping is a must.
Cowley Road can be a good way to see other parts of Oxford, but I wasn't too impressed with it. There are quite a lot of restaurants there, but the one we ended up in on my last visit was pretty underwhelming.
To escape the crowds, try Port Meadow along the river a bit upstream. I really enjoyed my visit there, a bit of peace and quiet as well as a nice landscape.
And if you want to see a completely different part of Oxford, you can visit the Mini plant.