The RS guide to England says there are two buses, #71 and #77, running between Terminal 5 and the center of Windsor on Peascod Street, which is very close to the castle. These run one to three times an hour, take 50 minutes, and cost 8 pounds (presumably one way). The guidebook also recommends Windsor Radio Cars, www.windsorcars.com, costing 25 pounds. It says the London black cabs charge whatever they like, presumably too much.
Also, the "England" board on this forum includes a question entitled "Getting from Windsor to Heathrow," last answered on 8/25 with links to Windsor Radio Cars and another car service. I can't vouch for them but you might check them out.
Your layover gives you plenty of time to get to Windsor, tour the castle, and return in time to check in at 8:00 for your 10:00 flight. I spent about three hours in the castle, including the state rooms and St. George's Chapel, and I took my time. (But I wasn't coming or going to/from Heathrow.) You'll need to put your luggage somewhere, hopefully at the airport. Maybe you can check it early with your connecting flight.
I suggest that you post your question on the "England" board where it's likely to attract more useful replies than under "Transportation."