First of all, I'd highly recommend getting another guide to supplement the Steves' guides. Unfortunately, the Steves' guides are very selective. They leave out a lot of really wonderful places to visit, and concentrate on selected sites. The Lonely Planet and Rough Guides are quite good, as is Footprint Scotland.
There are no direct ferries from Scotland to Ireland - only to Northern Ireland. There are ferries from Stranraer (Cairnryan) to Larne or Belfast, but if you want to go directly to the republic, you'd need to travel to Liverpool or Holyhead. You can fly directly to Dublin or Cork from most Scottish airports; however, beware of RyanAir.
You won't need a car in Edinburgh. The train doesn't go directly to St. Andrews. You'd need to get off at Leuchars and take the bus in from there. If you're going on to Inverness, you'd have to return to Leuchars, take the train to Aberdeen, then transfer there. Your other option would be to take the train from Leuchars to Perth via Ladybank, then taking the train north from Perth. You would need to change trains at Ladybank, and possibly at Perth.
For my money, I'd much rather drive. You can stop wherever and whenever you want, and your time is not controlled by the train schedule. It does take a couple of hours to a day to get used to driving on the left, and shifting with your left hand, but once you get the hang of it, you'll have a blast! You can rent your car at Edinburgh airport, once you've visited the city, and then drive on to St. Andrews from there. Don't limit yourselves to Europcar. Avis, Hertz, Enterprise, and National all offer competitive rates, especially if you book from the U.S. and use one of the many discounts on offer (AARP, Costco, Entertainment, etc.). Don't worry about not being able to drop off in a small town. You can drop off at a larger town, then take public transportation to your final destination.
You'll find that car hire is much higher in Ireland than in Scotland, due to mandatory insurance purchase.
Best wishes for a wonderful holiday!