Immigration (passport control), customs, and security are three different things, but people in the US often say "customs" to mean all of them. However, to avoid confusion, it's important to understand the differences. Customs is about goods, passport control is about people, and security can involve either one.
For your departure: When you check in at JFK, your checked bags will be tagged "AMS" (make sure of this). You will then not see them until you get to Amsterdam. You will then go through security, then to your gate. There is no customs or passport control at this stage.
In KEF you will go through immigration (your passport will be stamped). This is your entry into the Schengen zone, a group of countries that have agreed to abolish routine border checks (so, once you're in, you don't usually have to checked again as long as you stay in). Both Iceland and the Netherlands are in this zone.
You will then go to your next gate for your flight to AMS. Again, no customs. I don't know if there is security screening when changing planes at KEF (the layout in each airport is different, and I haven't used KEF).
In AMS, you will pick up your bags and go through customs. However, in Europe customs is merely a matter of walking through the Green Channel Nothing To Declare line. Unless you're "randomly selected for additional screening" (possible but rare), you're done, and are free to exit the airport.
For your return: When you check in in Amsterdam, your checked bags will be tagged "JFK" (again, make sure they are). You will then not see them until New York. You then go through security to get to your gate for your flight to KEF.
At KEF, you will go through passport control (this time it's emigration, as you are leaving the Schengen zone). Again, I don't know if there is security when changing planes there, but there is no customs.
At JFK, you will go through passport control, then pick up all your checked luggage, then go through customs. Here you have to hand a form to a customs officer (you'll get the form on the plane - keep a pen handy), and the officer will look at it. The officer may just wave you through, or may ask you some questions, or may pull you aside for further screening (once in San Francisco, they opened all my luggage and went through it thoroughly). Once you're done with this, you go through the opaque doors, and are officially "back" in the US.
EDIT: Cross posting with some of the above replies, so you see I repeated some things.