Gaston Paris / Contributor
A month old but still ignored study recently featured wired Mathematically provable claims that large language models are “unable to perform computational and agentic tasks beyond a certain complexity” – a level of complexity that is, crucially, very low.
The paper, which has not been peer-reviewed, was written by Vishal Sikka, former CTO of German software giant SAP, and his son Varin Sikka. Sikka Sr. knows a thing or two about AI: He studied under Turing Award-winning computer scientist John McCarthy, who virtually founded the entire field of artificial intelligence, and actually helped coin the term.
“There is no way they can be reliable,” Vishal Sikka said. wired.
When asked by the interviewer, Sikka also agreed that we should forget about AI agents running nuclear power plants and other obvious promises made by AI boosters.
Ignore the rhetoric from tech CEOs on stage and pay attention to what the researchers working for them are finding, and you’ll find that even the AI industry agrees that the technology’s architecture has some fundamental limitations. For example, in September, OpenAI scientists acknowledged that AI hallucinations, in which LLMs confidently make up facts, were still a widespread problem even in increasingly advanced systems, and that model accuracy would “never” reach 100 percent.
This seemed like it would make a huge dent in the viability of so-called AI agents, which are models designed to complete tasks autonomously without human intervention, and the industry universally decided last year that this would be its next big thing. Some companies that adopted AI agents to reduce their workforce quickly realized that those agents were no good at replacing outgoing humans, perhaps because they tended to hallucinate too much and could barely complete any tasks given to them.
AI leaders insist that strong guardrails outside AI models can filter out hallucinations. They may always be prone to hallucinations, but if these mistakes are rare enough, eventually companies will trust them enough to start performing tasks they once assigned to flesh-and-blood grunts. In the same paper in which the OpenAI researchers acknowledged that the models would never reach perfect accuracy, they also rejected the idea that hallucinations are “inevitable”, because LLMs “can go away if they are uncertain.” (Despite this, you’d be hard-pressed to find a single popular chatbot that actually does this, almost certainly because it would make chatbots seem less impressive and less engaging to use.)
Even though he is adamant that LLM has a strict limit, Sikka agrees with those in the AI industry who insist that hallucinations can be curbed.
“Our paper is saying that pure LLMs have this inherent limitation – but it’s also true that you can build components around LLMs that overcome those limitations,” he pointed out. wired.
More on AI: OnlyFans appears to have fallen prey to rival AI psychosis, which we dare try to explain to our parents
