From Edinburgh to Dumfries and the Garden of Cosmic Speculation

I am assuming that you mean by public transport

Normally the answer would be simple- train via Carlisle and Dumfries.

However on that Public Holiday Sunday all routes into Carlisle from the north are blocked with Engineering work.

So this is the only work round I can see-
Train Edinburgh to Glasgow Queen Street,
Coach-Bus # X74 at 0910 from Glasgow Buchanan Bus Station, arrive Dumfries Whitesands 1120 (these are nice long distance buses, usually in the Stagecoach inter urban bright yellow colour scheme).
Bus #246 at 1135 (headed for Cumnock), ask to get off at Portrack- this is 3 stops after a village called Holywood! (or the forum will doubtless say take a taxi, but all options are being offered to you).

Buses back are at 1635 and 1857 to Dumfries. Both arrive back too late for the last bus back to Glasgow.

So take the train at 1707 to Carlisle, and the rail replacement coach Carlisle to Edinburgh arriving at 2025.

Bluntly I would choose to book a B and B in Dumfries for the night and go back to Edinburgh next day. Or even for 2 nights- Saturday and Sunday. Dumfries is not an expensive place to stay.

Or rent a car for the day- Holywood is about a 2 hour drive from Edinburgh Sighthill or Airport if you rent there and is a very scenic drive, especially south of Abington services.

If you rent a car I would suggest having as early a start as possible then at Abington taking the B797 to Leadhills and Wanlockhead- very interesting places due to their gold and lead mining history- the two highest villages in Scotland and with the two oldest libraries in Scotland.
Then drive the Mennock Pass down to the A76 and straight down the A76 to Holywood. Another spectacular drive, which not many international visitors know about.

Then afterwards straight back to Abington over the A702.

The lack of public transport that day actually does you a great favour.

On the way back from Abington to Edinburgh give yourself a stop in Biggar- a town full of history.

Thank you isn31c. As we didn't buy parking at the garden, and have never driven on the left, we don't want to rent a car, tempting as your itinerary sounds. We may end up taking a cab. Or finding someone in Scotland to drive us. We do know a couple of people there.

For anyone looking at this in future years I would strongly suggest booking for the Saturday when public transport is way easier.

Or book 2 nights hotel in Dumfries, Thornhill or Sanquhar. In Dumfries a double room can easily be found for under £100 a night. Single rooms are not hard to find at £50 a night in good quality guest houses in Dumfries.