Thanks to everyone for the recommendations. They were incredibly helpful, and we visited many of the places mentioned. Rather than a full rundown, I thought I’d add a few additional notes that might be helpful to anyone who comes across this thread in the future.
We stayed at the Magnolia Hotel. Great location, very nice hotel with a very good restaurant. We’d happily stay and eat there again.
For a casual lunch with a view, we enjoyed Aura at the Inn at Laurel Point. The food was good, but the real attraction was sitting outside overlooking the harbor, watching the water taxis, seaplanes and other activity.
Q Bar at The Empress is worth considering for a cocktail. We weren’t particularly interested in afternoon tea at The Empress, and this was a nice way to experience the hotel and its atmosphere.
We visited Butchart Gardens and did afternoon tea there. The patio overlooking the gardens is a beautiful setting, and it costs considerably less than afternoon tea at the Empress. The food itself was just okay.
For anyone with an extra day, we drove into the Cowichan Valley and had lunch with wine at Unsworth Vineyards. The restaurant is new and attractive, overlooks the vineyard and restaurant garden, and the food was very good, including ingredients from the garden. We enjoyed the wine too, but I’d set expectations for a small, emerging wine region. One quirk: there’s a dairy farm across the road, so you may notice the smell outside. It wasn’t an issue inside the restaurant.
One completely non-food recommendation: the Victoria Bug Zoo was surprisingly fun. We’re not particularly into bugs, but visited when an excellent guide named Grey was showing them and letting visitors hold some. That made the experience, even if some of the holding was a bit outside our comfort zone!