A classic Victoria experience is to have high tea at the Empress: https://www.teaattheempress.com (Note: this is pricey, and I haven't done it myself, as I preferred to have curry in the Bengal Room, which is no longer an option, unfortunately.)
It's been a while, but years ago we really enjoyed Nautical Nellies: https://nauticalnelliesrestaurant.com
For a casual lunch, walk around the harbour to Fisherman's Wharf: https://www.fishermanswharfvic.ca We liked Barb's Fish and Chips: https://www.barbsfishandchips.com
For Indigenous food, there is the Songhees Food Truck at 1502 Admirals road. Again, this would be a more casual option, rather than fine dining, and I haven't tried it myself, as it wasn't open the last time I was there. https://www.songheesevents.ca/food-truck
Note that there is a little water taxi that can take you to different points around the harbour, if you need a break from walking.
And I know you didn't ask for activity suggestions, so I'm sorry if I'm overstepping, but I recommend:
Whale watching from a Zodiac with Prince of Whales. (We've gone whale watching 3 times, and our experience with Prince of Whales was the over-the-top best, although there are no guarantees with wild animals, of course.)
Fan Tan Alley is a cute little spot.
Visit Thunderbird Park, very close to the harbour, to see Totem Poles.
The Butchart Gardens are popular (need to book a bus or something to get there), but it's probably getting late in the season for it to be at its best.