tonfromleiden, Sam, and djp_syd have recommended Nuremberg as a base town for you. You asked, "for a first-time visitor to Germany who doesn't want to rush from one place to another, is a visit to one historic town and spending more time just exploring it a mistake? In other words, if I go to Nuremberg, would I miss much not going to Bamberg?"
With a toddler I'm sure your outings cannot be taken lightly. You probably want any outings you take to be fairly short and uncomplicated. Nuremberg is smaller than Munich of course but still large enough that your location will need to be well strategized. Hopefully you'll find a place near what you want to see and also near the station for outings. Bamberg is surely worthwhile but may not be that easy an outing as the station location means a long walk or bus ride into the old town. From N'berg you might find a smaller town like Iphofen, which Sam suggested, a much easier place to visit. It's gorgeous and just 45 minutes by direct train from Nuremberg. Who cares if it's not in RS's book?
http://tramino.s3.amazonaws.com/s/iphofen/685626/stadtplan-iphofen-english.pdf
Sam also suggested Bad Windsheim, I presume for the excellent open-land-museum there. This place and its 500-600+-year-old buildings really showcase local history. It should be pretty easy to visit too because it's family-oriented. They rent out wagons for wee ones that can't walk the whole path. Restrooms and eateries are on site as well as in town.
Map of the museum: http://freilandmuseum.de/fileadmin/bilder/C_Museumsplan2010_2MB_20110701153508.jpg
You can get there in less than 1 hour by train (1 change in Neustadt, a small station.) I suggest you inquire about an English-text guide to the museum when you arrive as written signage there is just in German as I recall.
"Also, in smaller towns, will we have problems not knowing how to speak German (I know most Germans speak English, but I didn't know whether that was limited to cities)?"
I've run into so many international travelers from Japan and elsewhere who speak no German and less English than you and still don't have much trouble getting around in small German towns - especially places where they get the occasional foreign tourist. You'll be OK. But yes, there is less English in the small towns than in the cities - but probably still more in Germany than in comparable small towns in other countries like France or Italy.
Someone above mentioned the Tagesticket Plus for train outings from Nuremberg. It's one of the best train deals anywhere. And if you buy it for a Saturday outing, it's good on Sunday too:
http://www.vgn.de/en/dayticket?&Edition=en&p=tagesticket-plus
Munich IMO is worthwhile, but skippable. A stay in Nuremberg will be different - but equally rewarding, I'm sure.