We found the SeaCity museum to be quite interesting in Southampton.
We've been to Portsmouth multiple times as my sister lives there. Only the one to time Southampton. I'm generally not a huge fan of naval history, but on our 5th visit to Portsmouth, we took the dive and did the Portsmouth Historic Dockyard - we lucked into a weekend online special on the tickets - I think it was buy one get one half off or buy one get one free (because with our Canadian dollar at a poor exchange rate, it had always been an expensive prospect). If you do decide to go, check into getting the tickets online - even if there is no special, there is still a 10 or 20% discount (or there used to be). We spent about a half day there - the highlight being the Mary Rose, which I guess is now even better as the Mary Rose is now not behind a big wall (it was drying out and still being preserved when we were there in 2015).
Depending on how many days, we also enjoyed visiting Arundel Castle, which was about an hour train ride from Portsmouth. I think you can stop there as well from London - I know we always pass by it when we train to Ports.
Edit to add - we also enjoyed visiting the Royal Garrison Church in Ports. Most likely because the custodians the day we were there were super chatty and very knowledgeable - I think we spent almost an hour there just hearing stories from the custodians...