I understand it's possible to do a day trip to St. Andrews from Edinburgh, but you'll need an early start and it will be a long day, whether by train or rental car. Since you also want to visit Dundee, yes I would say rent a car and plan 2 full days/2 nights.
If you rent from the airport you'll be driving on mostly highways, well marked for the Forth Road Bridge and onward up the M90. If you rent from Waverley Station you'll have (I think) a lot of stress navigating city streets.
St. Andrews walking tour of the Old Course was a true delight for my husband, a lifelong golfer. Our guide was wonderfully friendly. Book ahead for sure
Note that "walking tour" basically means standing around at various points near/on the 1st/18th holes. You do not walk the whole course, not by any means. What you do get is plenty of stories about famous golfers and what they did on various occasions in famous tournaments. It's heavy on golf jargon and delivered in a good Scottish accent, so for those who don't follow golf (or speak American) it can be pretty incomprehensible.
While you're there, you can have a nice lunch overlooking the Old Course in the British Golf Museum, and play mini golf on the "Himalayas" (ladies' putting green). It was a bit difficult to find the window where you pay and get your mini golf clubs & scorecard since the clubhouse has a big "Members Only" sign and of course we aren't members! There is no bag check or other place to leave your belongings -- I ended up playing while wearing a backpack.
I'd suggest staying 2 nights in St. Andrews and making a foray up to Dundee for a day; it's only about 1/2 hr to 1 hr drive, depending on traffic and where in Dundee you want to end up. The Old Course Hotel is the iconic place to stay, a little more expensive than some others but not terribly -- and it has an indoor swimming pool. If you'd rather stay someplace in between, Leuchars might work for you. Finally, if you're determined to stay in Edinburgh instead, you'd probably need to choose a hotel as near as possible to the airport for convenience in getting your rental car & returning it.