My daughter and I just spent 4 days in Edinburgh and I don't think we saw everything we wanted to see. We roamed around a lot, went to several museums, went into the Scottish Parliament, took a Mercat tour of the Underground Vaults, and one day took the bus out to Portobello.
Have seen the insides of enough castles, so didn't go into the one in Edinburgh, but would have liked to go into Holyrood, but it was closed while we were there. Walking up to the top of Arthurs Seat was planned, but we had quite a bit of rain, so didn't get to that sadly. Our B&B looked directly out onto it, so that was tempting.
It kind of depends on what else you are seeing or doing on your trip. I am the type of person who wants to stroll around a city, pop into places, discover restaurants and chat with people instead of rushing around from one tourist site to the next. There are some good tours in the city, a couple of which I wish we had taken. The Literary tour and one of the Mercat Ghost tours. We did go on the Sandemanns free tour and it was the biggest waste of time. Truly awful in my opinion.