A leading AI expert has warned that some Australians are showing signs of psychosis or mania in interactions with chatbots, arguing that Silicon Valley is being “reckless” with the technology in the pursuit of profit.
During an address at the National Press Club on Wednesday, Professor Toby Walsh, an artificial intelligence scientist at the University of New South Wales, said he believes the AI race will be both “boom and bust” with few benefits.
But his speech – a copy of which was provided to Guardian Australia – also warned about the dangers, which he said have loomed as the technology has matured in recent years.
“My childhood dreams are turning into reality, which is both good and bad,” he said in his prepared remarks.
Sign up: AU Breaking News Email
Walsh’s speech highlighted the legal case against OpenAI by the family of American teenager Adam Raine – as well as its data showing that more than one million of its users each week send messages that contain “clear indicators of possible suicidal planning or intent”.
OpenAI has also said that its number has been put at 560,000 800 million weekly users Symptoms of psychosis or mania have been reported, and another 1.2 million have developed potentially unhealthy relationships with chatbots.
Walsh said some of the people captured in the data were in Australia.
“I know because some of them or their loved ones have been contacting me by email,” he said in his prepared speech.
“They tell me how the chatbot confirms their wild theories. The chatbot tells them, to quote one email, that they have ‘cracked the code.’ That they are ‘the only one who can do it.’
Chatbots are designed that way, Walsh said.
“They are designed to be flattering. They are designed to confirm what the user says. And they are designed to engage the user. They always end with an open question, motivating you to continue the conversation and buy more tokens.”
He said it is not in the best interests of the companies responsible for chatbots to ask users to log off.
“There’s no reason they couldn’t be designed that way. Except that if there were careless people in Silicon Valley they would make less money.”
OpenAI has claimed that the GPT-5 update has reduced the number of unwanted behaviors from its product and improved user security.
Walsh also expressed anger at the “rampant theft” of creative works being used to train AI and the summarization of news articles in search that drives traffic away from news sites.
“Legally you can’t call it fair use when you’re competing with the owner of the IP,” he said.
“I refuse to accept an AI revolution that enriches founders in Silicon Valley while impoverishing Australian artists, writers and musicians.”
Walsh took aim at the companies, saying they were disregarding the laws, especially when it came to scams.
In November, Reuters reported that Meta’s internal documents at the end of 2024 said Meta was projected to earn about 10% of its total annual revenue – about $16 billion – from illegal advertising that year.
Meta responded by saying it had reduced scam ads by 58% over the past 18 months.
Walsh said that AI is being used to generate these scam ads, and Meta allows advertisers to use AI to manage these ad campaigns, while the AI decides which ads people will see.
He said that if 10% of a retailer’s goods in Australia were counterfeit or illegal, it would be closed by the weekend.
“So I don’t understand how we continue to have meta trading in Australia,” he said.
Walsh said he was disappointed that the Australian government was not doing more to regulate AI.
“I fear we are repeating the mistakes of social media,” he said. “Social media should have raised the alarm about the harms of unregulated AI.
“The harms we’ve seen from social media are about to be eliminated with even more powerful and persuasive technology.
“What I fear most is that I’ll be back here in three or four years and say: ‘We tried to warn you. But another generation of young Australians have now been sacrificed for the profits of big tech.’
