The key issue is the type of fabric that your clothes are made of. If you have the quick-drying synthetic tuff you can get at, say, REI, you should be OK. But if you have cotton clothes, then they'll take longer to dry no matter what. Jeans take an epic amount time to air-dry, so taking them along with that limitation in mind is a personal choice. I may try a "travel" pair from Travelsmith, since I like jeans too much to forego them.
I've heard of a trick where you roll your clothes in a towel to wick away excess moisture, and then hang them up to dry (without the towel, of course). I haven't tried it, but it may help them dry faster.