We have rented cars in Ireland five times (so far) and only had one "problem", and that was with the gate getting out of the rental yard (it wasn't opened all the way and I was seriously jetlagged).
If we're traveling with four adults and carry-on luggage, we rent a "compact SUV". If we're just a couple, we rent the smallest available. If you have large luggage, something like a RAV4 might be appropriate, though pricey.
Brand-wise, I don't think there's a lot of difference. I usually look on a consolidator site (like Momondo.com), look at the prices and book directly on the company website. A lot of people here like Celtic Legend as a consolidator and I think they offer different insurance options than the actual brands, but I've never used them. Be prepared for a small scrum at the counter over insurance, glass/tire coverage, fees to drive into Northern Ireland, etc. You can expect your rate to almost double if you say yes to everything. If you decline any coverages, make sure to point out pre-existing defects before driving away (don't forget rims/tires). We always use our credit card insurance coverage, but if you do, make sure you can produce an email from the card people explaining that you're covered in Ireland.
We've gone straight to Belfast after a longhaul flight from the west coast, and parked the car just off the A12 on Grosvenor, which allowed us to walk to our hotel. Not sure what the "scenic route" is, but the main road is "freeway" and the carpark is two blocks from the exit - no Belfast confusion. If you're arriving from the US, try to get some sleep on the plane! There's a toll plaza near the border, so have a 2 Euro coin or credit card handy - I had a 2 Pound coin all ready and got in the cash lane - whoops.
There are small roads (with stone walls hidden behind shrubbery) everywhere. Read about how to properly use "passing places", always stay on your side of the road (don't think you're being polite by pulling into a wide spot on the wrong side of the road) and don't get in a queue of cars longer than the passing spots. If you get behind a tractor or cattle/sheep, just relax and enjoy the slow drive. I don't think I've ever had to pull in the mirrors, even when passing a jumbo tour bus on a single tract road. Just pull over as far as you can without scratching the car, and stop until it resolves. We have had to back up to find a wide spot. On a famously narrow road, like Slea Head in Dingle, just join a small pack of cars doing the same thing - safety in numbers.