5 1/2 hour layover in Istanbul...

Yes, it is possible to leave. You just have to pay the $20 for the visa. Also, if doing this make sure your bags are checked all the way through so that you are not burdened with it. We did this 2 years ago - though we had about 7-8 hours. You can easily take the train & tram combo to get to Sultanhamet within 30 mins (prob 1 hr from disembarking). This should give you 3 or so hrs to walk around. Since you should have your onward flight boarding pass, you just need to get back to the airport about 45-60 mins before departure. (I'm sure others will tell you to get there 90 or more mins ahead of time).

I think I'd recommend hiring a guide to meet you at the airport and drive you into the city, drop you off for a bit of sightseeing in Sultanahmet, and then pick you back up and return you to the airport. 5 hours is just a bit tight for trying to get into town on your own, or at least it would be for me.

Definitely go into the city! Take the metro (1.75 Turkish lira = around $1.25) to the tram (another 1.75 TL) to the Sultanahmet stop: travel alone won't be more than 45 minutes. and you can see the Blue Mosque (free, allow maybe 20 min), Basilica Cistern (10 TL, allow 20 min), Agia Sofia (20 TL, needs an hour or more), or Topkapi Palace (20 TL + 15 for the harem, and you can easily spend 3 hours there).

I would suggest you to take havas bus or taxi and drive to SA. It's quite a short time to visit some places of interest and explore them in full, so I recommend to go to a Turkish restaurant - Pierre Loti, Tria Elegance, Abracadabra - there are plenty of them. It's a great alternative for a short period of time in Istanbul.

I would skip Topkapi Palace, given your timeframe, as it is a major site with several buildings. Aya Sofya is a must see, as is the Blue Mosque. The Underground Cistern is also striking, but I might opt instead for a quick visit to the Grand Bazaar or the Spice Market. The important thing to keep in mind is that you need to budget time for getting from place to place, starting at the airport. It will likely be a good hour from the time your plane lands till you are deplaned, through the line to buy your visa (don't forget your $20 US bill!) and get your passport stamped. By metro and tram it's easily another hour,and an hour back, too. (That's why I would recommend arranging for a driver, who can travel the same distance in less time. Depending on traffic, it should not take more than 40 minutes.) And you need to allow at least another hour and a half at the airport to get back through security to your connecting flight. Whatever you do, eat something before you head back to the airport. In a world of ripoff airport food, Ataturk Airport is notably bad.

agree you should skip topkapi palace but instead focusing on hagia sofia and blue mosque. i personally really like the inside of hagia sofia just a lot of different layers of history and culture, and the line is shorter because you have to pay. the blue mosque is free but there is a line you wait to get in. The grand baza is on the train line to/from airport, easy stop for a quick walk.

I was in Istanbul in September and think trying to see anything in 5-1/2 hours is tight. I took the train/tram between the airport and Sultanhamet, and is was closer to 1 hour each way. A cab is no faster due to heavy traffic all hours of the day. You may have a wait to get a visa - it took me almost 1 hour from landing to exit the airport. You'll also need to allow time to get through security/immigration and to your gate once you're back at the airport. Anticipate that you'll have 1 hour, 1-1/2 hours max, in Sultanhamet on a 5-1/2 hour layover. You should also consider season and time of day. Sights were switching to winter hours when I was leaving, which means they close about 4:30pm with last entrance about an hour earlier.