International drop-off fees can be very high, but what you've found is unusually dreadful. If you've checked with just one rental agency, you might try going to the consolidators (AutoEurope, Kemwel and Gemut) to see what they quote. I doubt that you'll find it acceptable, but you never know.
The problem with dropping a car near the border and picking up a bus into Spain is that there is very little public transportation between the two countries. I think there's only one train a day between Lisbon and Madrid, and I believe it runs overnight. As of 2015 there was bus service from Badajoz to Lisbon, but I don't know at what frequency, and that's south of where you'll be. There's some service down in the Algarve (even worse), and there are both buses and trains running from Porto north toward Santiago de Compostela (too far north). None of those sound very attractive to me, but there may be some other bus options that I'm not aware of.
I'd start with your planned driving route, noting the larger cities near it. It looks like your best bets are Porto itself, Vila Real and Braganca, which are reasonably near the Douro Valley. Farther south you have Aveiro, Viseu and Guarda. Then go to Rome2Rio.com to see whether there's bus service between any of those spots and Madrid.
I'm not too optimistic, because I suspect that the territory near your two potential border-crossing spots is rather lightly populated, which makes the existence of buses in those areas rather unlikely.
I think your best bet will be to fly from Porto to Madrid. There definitely are flights. Check Skyscanner for details.
Oh, one other possibility: A ride-sharing service (think formalized hitchhiking) like blablacar.
