Apart from the Cathedral there is also Ely Museum plus Oliver Cromwell's House- you can buy a combo ticket for those, the Cathedral and the Stained Glass Museum, but only at Oliver Cromwell's House, as far as I am aware.
Another hidden gem is Bishop Crauden's Chapel, in the old monastic close- you are meant to make prior arrangements to visit, although in term time you can in theory turn up to the Kings School reception and ask for the key.
Afterwards stroll down through the meadows to The Maltings, at the side of the River Ouse- it is quite lovely down there (a quintessential English riverside), and is also the nicest route to or from the Railway Station, which is at the bottom end of town (whereas buses stop on Market Street in the Town Centre, saving the walk up from the station). From Market Street the obvious approach to the Cathedral is to the Almonry then the side gate into the Cathedral close.