Sounds exactly what my wife and I did two weeks ago. Got into CT around 5pm and then stayed two more full days, before early train to Pisa Airport following morning. We agree that was our favorite spot ever . Some people seem to think a couple of days is a long enough time, we disagree and already have reservations to go back for a longer time next year. My advice to you - get up early and stay late. The towns are magical in the early morning and evening (at least Manarola was). Hit the trails early, they are quite thin in spots. Yeah the towns are crowded, even in midweek September from noon to five. My advice is settle in for a long relaxing lunch somewhere around 1 pm. Have a few courses and a bottle (each) of CT wine. Then just go out and mix with the masses for a few hours before they hop back on the ferry.
Since you will have only a short period of time, i'd really try to manage your train travel wisely. The only "problem" we had in Italy was trying to buy train tickets. You'll find long lines where you can buy tickets from humans, and a sea of confusion at the ticket machines, and there are never enough machines. At the machines there will be confused people taking fifteen minutes, malfunctioning machines, and machines taking your order for train lines that aren't for sale on those machines. My wife and I just jumped on the train, twice, after making attempts to purchase them, and never ended up paying. Karma caught up with us when we actually did buy tickets to go from Manarola to Monterosso and went in the wrong direction. We got off at La Spezia and jumped on the right train. When we explained to the conductor that we simply got on the wrong train, we were scolded for not buying new tickets and fined ten euros each. I now laugh when I hear people say "simply buy tickets at the station".
I'd be happy in any town, and I'd base my decision on the lodging and water view. But Manarola is now our town and that's where we are going back.
