I went swimming at a public pool in Reykjavik on a long (12+ hrs) layover several years ago. I think it was Sundholl (looking at the photos, I believe that's the one. It was an outdoor pool). I was there during a lap swim time and thoroughly enjoyed swimming a few laps to stretch out after sitting on the plane for several hours, and before getting back on a plane for several more. There might have been part of the pool open for general swim/free swim, but I can't recall for sure as once I saw the lap swimmers, I just joined there.
Do follow the directions posted in the locker room for pre-pool cleansing/showering.
Make sure your swim suit is packed in your carry-on (if you're also checking a bag if on a layover). Also if on a layover consider having a change of clothes or at least underwear in your carry-on. After the relaxing and refreshing swim, I found putting on the clothes I'd been wearing for many, many, many hours before, a little gross (especially knowing I had so many more hours ahead of me). When I did something similar on my return trip, I made sure I had a fresh top and underwear to change into in my carry-on.
Trying to recall what I did for a towel, I think I must've had a packable towel in my backpack.