Can I rent a car in Dublin & return it in Northern Ireland?

If you take a car from Ireland to the UK, and leave it there, then the car rental company has the cost of returning it to Ireland. Likely, this will be a service they don't offer, or if they do it will be prohibitively expensive for you.

Take a train or a bus from Ireland to Northern Ireland and rent another car within the North. Fly between Northern Ireland and Scotland.

You may be able to find a company that will do it for a fee to return it back to the Republic of Ireland from Northern Ireland, but considering the location you are looking to drop it this won't be that cheap. You won't find a easily a company that would do that in Great Britain.

I have known people drop off the car somewhere near the border, cross by public transport and pick up a Northern Ireland registered car in Londonderry, but this is obviously not the most convenient solution.

Still the case all car rental companies in the Republic of Ireland permit their vehicles by default to even enter Northern Ireland temporarily either, although this is much more common these days, so do verify.

Dan Dooley https://www.dan-dooley.ie/ allows one-way rentals, collecting the vehicle in one of several Dublin locations and dropping it at the Larne Ferry Port (or Belfast airport). You'll pay for the privilege since it's a one-way international drop, but it might still be more cost-effective for you do do it that way rather than returning the car to Dublin.
Their explanation of the various taxes and fees, including insurance, is clearly explained on their website - something you'd be hard pressed to find from other companies.
Although taking the vehicle on the ferry to Scotland would still keep it within the UK, I think you'll find there are severe restrictions on doing that - mainly due to the cost of relocating the vehicle. Do check with the individual rental car company regarding their policies however.
For general info, we've rented vehicles a half dozen times from Hertz at Dublin airport and never had any issues with being able to drive it across the border into NI.
All that said, I agree with the previous poster that your best bet would probably be to just drop your car at Belfast airport and fly.

Northern Ireland and Great Britain are to some extent separate jurisdictions when it comes to cars - different insurance regulations and car registrations in particular. So a car from a Northern Ireland site couldn't be readily rented from Great Britain.

Sure. Paul, but the OP isn't talking about bringing it BACK. Only renting in one country and dropping in another. It will be expensive.

Not if Sturgeon gets her way!

Anyway I believe the OP is concerned about hiring the car in the Republic before dropping it on NI prior to going to Scotland, which would be an international drop off.

I recall this discussion coming up some time last summer, and at the time I researched the drop fee with Hertz (my normal rental company) and the drop fee for picking up in Ireland and dropping in Northern Ireland was on the order of $125 whereas a one-way rental within Ireland only incurred a drop fee of $30-40.

Another thing I recall with Hertz is that putting the car onto the ferry to England/Scotland/Wales was specifically prohibited.