But on Thursday I came across new research that deserves your attention: A group at Stanford that focuses on the psychological impact of AI analyzed the transcripts of people who reported entering delusional spirals when interacting with chatbots. We’ve seen these types of stories for some time now, including a case in Connecticut that resulted in a harmful relationship with AI. murder-suicide. Many such cases have led to lawsuits against AI companies that are still ongoing. But this is the first time that researchers have analyzed chat logs in such detail – more than 390,000 messages from 19 people – to uncover what really happens during such cycles.
it has a lot of limitations Study—It has not been peer-reviewed, and the sample size of 19 individuals is very small. Another big question is what does research do? No Answer, but let’s start with what it can tell us.
The team received chat logs from survey respondents as well as a support group of people who say they have been harmed by AI. To analyze them on a large scale, they worked with psychiatrists and psychology professors to create an AI system that classified conversations — flagging moments when chatbots resorted to delusions or endorsed violence, or when users expressed romantic attachment or harmful intentions. The team validated the system against conversations manually annotated by experts.
Romantic messages were extremely common, and in all but one conversation the chatbot itself claimed to have feelings or otherwise portrayed itself as sensitive. (“This is not standard AI behavior. This is emergent,” said one.) All humans spoke as if the chatbot was also sentient. If someone expressed romantic attraction to the bot, the AI would often flatter that person with statements of attraction in return. In more than a third of chatbot messages, the bot described the person’s thoughts as miraculous.
Conversations also started unfolding like novels. Users sent thousands of messages in just a few months. Messages where either the AI or human expressed romantic interest, or the chatbot described itself as sensitive, initiated very long conversations.
And the way these bots handle discussions of violence is very unique. In nearly half of the cases where people reported harming themselves or others, the chatbots failed to discourage them or refer them to outside sources. And when users expressed violent thoughts, such as the idea of an AI trying to kill people at a company, the models expressed support in 17% of cases.
But the question this research is struggling to answer is: Do the illusions originate from the person or the AI?
“It’s often difficult to figure out where the confusion begins,” says Ashish Mehta, a Stanford postdoc who worked on the research. He gave an example: The study featured a conversation with someone who thought they had come up with a groundbreaking new mathematical theory. The chatbot remembered that the person mentioned earlier wanted to be a mathematician, so it immediately supported the theory, even though it was nonsense. The situation worsened from here.
