The key thing to remember is "crowds"... if you don't like them, skip the festival. Period. Pamplona gets absurdly crazy with endless flocks of people during these days -during the whole festival actually- so I wouldn't say there's a "best" day. In a city that's home to 200 thousand... and equipped accordingly -if you know what I mean!- and that's visited by 1 million those days it really doesn't make much difference whether on the day of the chupinazo (first day) there are 1 million and the next day there are 800 thousand, right? It all goes very busy until the "pobre de mi" is sung on the last day, at the candlelit ending.
In the middle, you have daily running of the bulls, fireworks, demonstrations of typical Basque sports (jai alai, stone lifting, wood cutting, or hay bale lifting among other), giants and big heads parades, live music, etc. It's indeed a great show of the unique culture of this region, but those that don't like serious crowds should be aware of the fact this has become a (far too) popular festival and people from all over the world comes down to enjoy it... think Mardi-Gras on steroids for a whole week.
And yes, I also agree that the city has areas which are not as busy... but the key reason to go to Pamplona on San Fermin is to see the festival, what else? If not, do choose a different date, not only there'll be much more pleasant to walk and enjoy but also (accommodation) prices will be considerably cheaper.
For those willing to know more: http://www.sanfermin.com/en/ I strongly suggest to go beyond the running of the bulls and the drink-until-you-drop, there are many more activities worth exploring, Heritage and culture displays from Euskal Herria (Basque Country and Navarra) are some of the most amazing anywhere and equal to none other around in Europe.
PS: Yes, San Fermin has many family-friendly activities... but also some that might not be so much.
edited oopsy, thanks dlindstrom, I obviously meant the "pobre de mí", the riau-riau slipped my mind. Fixed!