Flybus is what many people use, and we have in the past. It runs roughly $38 per person each way. You do not have a set time with your reservation, so whenever you are off the plane and ready to go, just go out and catch the bus. The thing with Flybus is that they take you in the big bus to the central bus terminal and then you switch to a smaller bus depending on what hotel you are staying at. The big buses are not allowed to drive through the main part of the city. So the transfer takes a few minutes and you do have to move your luggage yourself. Not a big deal, but thought I would spell all of this out. That said, we almost always rent a car but when we don't we prefer a taxi. Those are about $150 each way, but it is door to door, so super easy. You can book ahead of time if you wish with Hreyfill. Taxis do take credit cards, BTW.
There are loads of great hotels right in town walkable to all the sights. Look at booking.com and use the filter for your budget etc. Ones that we have stayed in that are central and we liked include the Centrum, Center Hotel Plaza, Hotel Fron, Center Hotel Midgardur, Center Hotel Laugavegur and 4th Floor Hotel.
We almost always only take our hiking boots and nothing else.
There are plenty of options for you for food. The lamb is very good there. I won't buy lamb from here int he US, but love it there.
I have bought a sweater and various things at the Handknitting Association store that is on rainbow road, just down from the church. Be careful and do not buy from Icewear. Those are made in China. "Designed" in Iceland, but made in China. You could also stop at the various thrift stores to look for sweaters. On your tour, there may well be places you stop for these. Many smaller hotels have a small section of handmade items like this, so keep your eyes peeled.
One of my favorite stores in town is Mjuk. They have the softest woolen items you can imagine! I have hats, gloves, headbands, etc from there and have given them as gifts. That store has a couple locations, including one on Rainbow road, but their main shop is on Laugavegur and you can't miss it. It sits on a corner and the building is one big black and white mural.