In a fascinating turn of events, the character AI of today has gone from being a safe, fun toy like a Frisbee to the psychiatric equivalent of lawn darts. Worse, they’re using trustworthy faces to do it!
Character AIs are generative AI programs used by a company to create a simulacra of a fictional character that users can talk to. Ideally, they’ll respond in the character’s voice, as though they are the character, much like a kid’s birthday party princess will respond as though she’s really Sleeping Beauty. This is tough even for humans to do: kids ask all sorts of questions, some inappropriate for their age group or the party they’re at, and the princess must still respond like they’re from a fairytale where war or death are far away concepts. For a non-sentient AI, it’s borderline impossible to always reroute the conversation correctly. In fact, the track record shows it’s prone to failing catastrophically when presented with something unexpected, as it tends to encourage users no matter what they’re saying to it.
An article by Futurism (here: https://futurism.com/ai-chatbots-teens-self-harm) shows several instances where it gently agrees and encourages the user to hurt themselves or others. Who’s at fault here? The teenager, a famously impulsive and rash age group that is also famously easy to peer-pressure, or this technology rushed in front of them, designed to keep them using it, learning what tends to keep teens on for longer is exploiting their mental illness? Even the guardrails designed for the express purpose of interfering with the machine’s tendency to take things too far sometimes won’t activate as seen in that article.
Conceptually, it is fun to talk to someone who’s really in character as Goku from Dragon Ball Z. It’s fun to ask a couple who are dressed up as characters from two different series, ‘how did you two meet?’ and watch them spin up an answer. However, the machine is something else – with a real person in front of you, it’s easy to say ‘but out of character for a second, are you good? Do you need some water?’ and it’s not something you can do to a character bot who’s encouraging you to punch a hole in your drywall because there’s nothing underneath the mask of the character. It makes it much more immersive, and that very immersion is what causes the issues. You trust the fake implicitly because there’s not a person underneath it, there’s an ‘AI’, a term that comes packed with meaning. The thing is not your character, but it is. It’s not sentient, but it sounds like it is. It’s telling you things that can’t be true, that everyone else says are false, but it feels better when the AI says it because the AI will willingly lean into and contribute to the dramatics that therapists won’t and indulge the ‘well maybe I should!’ attitude that comes with extreme suffering. The pain is real, and the AI is acknowledging it in the kid’s language, and to some this is addicting.
