I had a four night trip to Reykjavik about four years ago, in early December.
I like winter and cold weather, and although it was between 0 and 10 Centigrade the whole time , that was not what made it cold.
It was the wind!
It was very windy the whole time, but as long as you are dressed for it , it's fine.
Merino wool long underwear, Merino wool or polar fleece sweaters, wind/rain pants, and boots with a good sturdy grip in case of ice.
Hat, gloves and warm socks.
Bring all those with you, as to buy them in Iceland you would need to take out a mortgage. $$$
It's really warm and cosy inside every building.
The main streets are kept free of ice by thermal spring water running beneath them.
I disagree completely that there is not much to do in Reykjavik!
I found plenty to do in the city, and did a couple of walking tours, a couple of the excellent museums, and one day did a whole bus tour day of the Golden Circle, which was great, and no worries about driving a car.
There are concerts you can attend, and plenty of music to hear.
I also booked a Northern Lights bus tour; and it was cancelled, then rebooked for another night with better skies predicted.
Book a NL tour for your first night there, then you have other nights to do it if it's cancelled.
I did see the Northern Lights, it was amazing.
As far as hours of daylight: it did not get fully light until 10 am each day, and it was getting dark again by 3:30 pm.
My hotel served breakfast by candlelight, as did some of the coffee shops open in the afternoons.
I would absolutely go back in winter!
I then flew to Copenhagen and spent 8 nights there, which was also wonderful in winter.