OK to rent a car in Ireland, take it into N Ireland, and drop it off there?

You can do this if you are prepared to pay the likely fee for a one way rental. The vehicles can't be rehired locally and have to be taken back to the country of origin. Animosity doesn't come into this, just business.

Your pick up point in Cairnryan Scotland is at the port for the ferry from Belfast.

With regards to the Ireland question, it wouldn't be "animosity" but it would be the fact that it is a different country, and car rental firms generally charge more to drop across the border from where you rented. It may be simple to train across, for example if you are in Dublin (?) you can catch a two hour ride to Belfast, and rent a car there. Check out Kayak or Auto Europe and run some price examples.
The animosity you refer to is still present, but it is represented by the extreme factions of each "side", no surprise there.
Have a great trip! Ireland, and Northern Ireland, are wonderful, welcoming countries. Take a Black Taxi Tour in Belfast for a great primer in "The Troubles".

You might want to rethink the whole sequence once you compare the cost of dropping the car in Northern Ireland.

The Belfast to Cairnryan run is a bit less than a couple of hours.
So is the drive from Belfast to Dublin.

Stranraer/Cairnryan isn't close to much of anything of interest to the casual tourist.
You'd have to drive a couple of hours to get to Glasgow and about three to Edinburgh.
The flight to either from Dublin is about an hour. There's no customs/immigration between the destinations.

Using budget airlines, the airfare is going to be about the same as the ferry cost.

Maybe if you put up your itinerary it would help make alternative suggestions with better logistics.

These are the transport resources I use most: www.seaview.co.uk for ferry options, Auto Europe for car rental, and www.skyscanner.com for flights within Europe. Compare car rental price quotes to see how much dropping in Northern Ireland adds to the rental rate. I would not oppose Gail's suggestion to fly from Shannon airport to Scotland, if you decide either to skip Northern Ireland or to do it first and then loop southwest along the coast.

If you want someone to help you plan this on a calendar (but not to book reservations), you can also consider a paid consultation.