There are a few blocks where there is a large social services building called ARCH which draws homeless people (it's highly concentrated). Nothing unexpected or worth highlighting or getting freaked about (unless you've never seen a large homeless population before). It's just like any other large city. Of course the folks staying in high end hotels practically next door would be shocked, but it's part of the urban fabric. I wouldn't use the word "charming" to describe any downtown, including Austin (although there is a nice drag on Congress Ave with the old Paramount Theater and other historical buildings leading up to the Texas State Capitol, which is nice). The one thing that does really stand out in Austin is the youthful vibe (50k students at UT Austin). I'm in my 40s and can feel "old" in Austin because there are so many young, college-aged people seemingly everywhere (you'll rarely see many folks in the upper parts of the age distribution, at least out on the streets). There's also an eccentricity about the place and residents that's more like Venice Beach, CA. The music and food scene is great (especially the tacos, have mercy!), as as the greenways for biking and walking (very accessible from downtown). There are funky corridors like South Congress with thrift stores, hotels, food trucks, etc. The Hill Country outside of Austin is beautiful and scenic, one of my favorite places in the US.
I think Austin is great and worth checking out. But not if you're looking for a cultural scene like Houston or Dallas with museums, symphonies, etc. The Umlauf Sculpture Garden is a real gem in Zilker Park. If for some reason Austin isn't to your liking, you can easily drive out to see the Hill Country towns. You really can't go wrong by giving it a chance. The hotels downtown are insanely expensive during busy months (like during SXSW in March or Austin City Limits in October) , and there always seem to be festivals and events going most of the year. That's what makes Austin "Austin" - the annual events, music, film festivals, other performances, etc. I stayed downtown for work several years ago and then slightly outside of Austin for leisure trips (can't stomach paying north of $300 for a hotel).
This is a very good planning resource: https://www.austintexas.org