I have spent extensive time in both cities and it really depends on what you want.
Eger is small and quaint with a great castle hill, the lyceum, and two beautiful cathedrals. I've spent weeks there at an annual conference over the past 4 years. I really enjoy it.
Why not do 2 days in Budapest and see if you are able to get to all the sites that you mostly want to see. If so, then go to Eger as a day trip on the 3rd day (or stay in Budapest and see more sites if you feel you want more time there). No need to spend a night there, it's no hot spot. A well planned day would be enough. It will be a nice little relaxing reprieve. Or you could do it as a day trip on your way to Budapest.
But do make sure you go to the baths in Budapest. That's my favorite thing to do!