Yes. The Tower of London is absolutely worth seeing. In my opinion, it is the most interesting sight to see in London.
It is number 1 on my "must see" London list. #2 is Westminster Abbey.
When I was in London this past spring, I spent most of one day at The Tower. It takes a while to go through and see the Jewels. Do that first; there can be quite a line later in the day. It takes awhile to go through The White Tower, but it is fabulous. Lots of suits of armor on display. Even a suit of armor that belonged to the rather large Henry the 8th!
They have a great lunch room, with good salads, sandwiches, main dishes and desserts, The Armouries Cafe. We ate a late lunch there, rested, and had a cup of coffee before heading out to look around more.
The Tower of London is quite interesting. If you are absolutely pressed for time during your visit there, you can skip the Crown Jewels exhibit. It can get crowded in the afternoons, and in my opinion, was NOT the highlight of my visit. Your two boys will probably be bored with it.
DO make sure you explore the outer walls, and walk upstairs--up the winding stone staircases--and see the view of the river out the Tower windows. Walk some of the outer walls.
If you have any questions about how to get back to Victoria Station, just ask the employees at the Tower for advice. They are all very friendly and helpful.
Likewise, do not hesitate to ask the employees at the Tower Hill tube station for directions, or where to change trains, etc. They are very helpful. Right inside the tube station entrance on the left should be a booth with an employee, or one standing nearby.