Tory Island or Dunfanaghy?

I can only speak for Dunfanaghy. We used it as a base to explore that part of Donegal for 5 or 6 days a few years ago and enjoyed it very much. It's true that it's an understated little village but we found that to be much of its charm.
The day trip down to Glenveagh is worth doing. The old castle is a fascinating place with an interesting story attached to it - it was the site of the last great land eviction back in the 1860's. There's a very pretty walk along the river for a couple of miles to the head of the glen. Can pick up the fixings for a picnic lunch at the little cafe inside the castle.
From Dunfanaghy there are a number of other lesser known attractions and some exceptionally pretty drives along the coast, and Derry and the Ulster-American Folk Park near Omagh are easy day trips away by car. We didn't run out of things to do at any rate, and we found plenty of atmospheric little places for lunch and dinner.
Tory Island does look like a cool place but it also appears to take some getting to. I see that the ferry is pedestrian only which means that you'd be wasting a couple of your car rental days if that's important to you.

Thanks for your nice summary. I'll stick with my current plan and defer Tory Island for some future trip. I missed the castle (unless you mean Doe castle) in my planning, so I'll have to look at that.

Although I'm probably a bigger cheapskate than most, parking a rental car for a day or two of "waste" is easily justified with bus/train/taxi fares, especially if you are a party of four (which we usually are). Of course that assumes one's paying $35/day, not the rates of last summer! Oh, the fond memories of $120/week!

Might add that there are some very pretty gardens adjacent to the castle, which should be in full bloom by the time of your proposed visit.

Doe Castle, or somewhere else?

Glenveagh Castle.
It's website is really pretty good - featuring lots of photos and descriptive material about the history of the place.

I've got 3 1/2 hours in the park including lunch in the café, so visiting the castle should fit it, depending on how glorious the weather is for walking. Didn't realize there was a castle or garden. Thanks!