I think much depends on what aspects of Iceland and its history you're most interested in. I loved the Saga museum, which shows realistic people of old Iceland involved in vignettes of their personal stories. I think kids would like it a lot. I also loved the National Gallery, although that's a hit-or-miss proposition, since they carry different installations, but their core pieces are lovely; Iceland's artwork is very much a product of the "ljus" Danish school, and should be carefully researched before you see it, otherwise you'll miss the point of it.
The Maritime museum is also a personal favorite, especially if you're interested in the history of fishing (the industry that kept Iceland alive until relatively recently). Lots of poignant stories in that museum, since fishing in Icelandic waters was, and probably is, an extraordinarily dangerous job, as the museum makes very clear. Close to the Maritime museum is the Aurora exhibit, where I went one night at about 10 p.m. and had the place to myself. I learned a lot about auroras, and in fact, the exhibit explained so much about auroras, and put you in the middle of what it would be like to see one, that I finally understood why I have never seen an aurora in Iceland.
There are a few things that aren't necessarily fascinating, and yet should be seen if you're really interested in Icelandic history (and to an extent, world history when it comes to the Culture house). The Settlement exhibit is absolutely fascinating if you would like to see the bare bones of a farmhouse unearthed during building of what has become the older part of downtown. The exhibit is underneath a hotel, but has its own entrance. Culture house is where you must go if you want to see some of the earliest versions of the written 'sagas,' but there is so much more there, look it up online before you go, so you don't miss anything.
As for the old harbour, where the various whale excursions start, there's a lot to see and do right in that one area, as well as many fish restaurants. From there, you can walk (you have to cross streets with traffic lights) up and down the waterfront, walk down to Harpa, which is a must-see, if only from the outside, especially at night, when its 'fish scales' change colors all night long.
I just think that no matter what you might be interested in, look it up online before you go, because there are usually details you won't appreciate unless you've learned a bit about Iceland ahead of time.