Edward VIII

Doesn't seem like it.
From the website........

Frogmore House and Gardens are only open to individuals on three Charity Open Days each year, when all proceeds are donated to specially selected charities. Dates for 2020 have not yet been announced.

Frogmore is also open to pre-booked groups of 15 people or more during August only.

Then it says elsewhere that the mausoleum is never open to the public. But they're buried outside maybe?

This is what Wikipedia says:

The Duchess of Kent's Mausoleum Frogmore House and its gardens are
usually open to the public on about six days each year, usually around
Easter and the August Bank Holiday.

The Royal Burial Ground may be viewed from around its perimeter on the
days on which the gardens are open to the public. The Duchess of
Kent's mausoleum may also be viewed externally, but is never open to
the public.

The Royal Mausoleum, the resting place of Queen Victoria and her
husband/consort Prince Albert, is structurally unsound and has been
closed to the public since 2007. Repairs may take up to a decade to
complete.[3] The Royal Mausoleum used to open on the Wednesday nearest
Queen Victoria's birthday, 24 May and occasionally on other days when
the grounds were open.

The restoration project was begun in June 2018 and will protect the
Mausoleum from the historic damp problems.[4]

The Duke and Duchess of Windsor are buried in the Royal Burial Grounds of Frogmore, which surround the mausoleum, and not in the mausoleum itself. Not that it makes any practical difference for the OP's question. Assuming you visit Frogmore on one of the rare public open days, you can only look at the royal graves from the perimeter of the burial area, as I understand it.