I think people may have 1980s perceptions of downtown Los Angeles. I can't claim what it looks like right now (during Covid) but I lived in SoCal for 20+ years, visit regularly (have family there) and have been in various parts of downtown regularly and saw no issues. Obviously Skid Row is not nice but it's not like ALL downtown is some hell by any stretch of the imagination. Yes, there are homeless people but they are everywhere in Santa Monica too, which is a super wealthy enclave where one wins the lottery by scoring a rent control apartment. Personally, I prefer the beach by a long stretch, so I would stay in Santa Monica. The public transit isn't what it used to be either - it's gotten a hell of a lot better. Metro was extended to Santa Monica some time ago. Everyone drives in CA so they have no idea what public transit even exists. There is decent public transit in some areas (mostly the Blue Bus in Santa Monica or where students live, etc).
Bashing LA is an Olympic sized sport, so just take that into some consideration. Again, yes there are homeless people (lots and lots)...but there are so really cool parts of downtown to explore. Personally I think you'll be fine by Union Station but you don't need 2 days there. Unfortunately some parts of downtown worth checking out are not adjacent but cut off from each other by freeways, and navigating from one to another is not fun at all on foot. I try not to go under any freeway underpass unless there's a lot of foot traffic and in daylight - it's just creepy and, yes, homeless do congregate there.
Don't sweat it, you'll be fine in Chinatown/ Olvera/ Union Station area. I would ask your friends to hang out in Santa Monica for a while and then drop you off directly in Chinatown afterwards.