I might not be very much help. I went, I stayed a week, I got rained on. I happened to have been there in March, which I wouldn't really do again. I stopped at the tourist office, and we joked that I was their only tourist. So, here is what I can tell you about what I did:
My airbnb was in the LesButtes area - which is gorgeous and the neighbors were all so friendly. I'd not mind staying out there again, but at the end of the day, without a car, I did feel quite stranded. I had thought I'd be able to bike around, but I freaked out because the roads were so narrow and two-way. (I live on a car-free island in Michigan, US, and this was just not comfortable to me). So, I learned the bus system. From where I was staying, the more frequent bus took about 45 minutes to get into Saint Peter Port.
Saint Peter Port is a nice little city and I enjoyed walking around - but again, it was March, and it rained a lot. I tend to gravitate towards any town's "Old Town" but this was just a series of empty buildings. I walked around a bit but didn't see anything of interest, or anything open. Maybe in different months it would have been different. The Victor Hugo house happened to be closed when I was there, for renovation work, so that was out... though I did go to see the outside. There is a nice High Street, with some shops and restaurants. The parks are nice, and I did enjoy that. I stayed for one night downtown at The Duke of Richmond Hotel, which I highly recommend. I also took a spa day at the Old Government House, which was nice and relaxing. The castle was closed. The lovely staff at the Duke of Richmond helped me to book a taxi tour of the island, and my driver was a life-long islander who knew all of the best spots to drive me to, explained the history and how it even tied into his family. I am so glad I did this, even though it was raining almost the whole time. The whole tour took 2 hours. Someone I met on the ferry from Poole had convinced me to visit Jersey, so I flew to Jersey for one night, and it was GREAT. I had intended to visit Herm or Sark, but was told ferries were not dependable at that time of the year, so I opted for this option. I enjoyed my one night/day on Jersey, then returned to my little airbnb on Guernsey for my last night. So, as I said, I might not be much help, but am glad to share my experience.