Unlock Editor’s Digest for free
FT editor Roula Khalaf selects her favorite stories in this weekly newspaper.
Elon Musk’s artificial intelligence chatbot has created sexually explicit images of children that have been shared on social media platform X, raising concerns about the safety of the model used by millions of people.
Over the past few days, users have been able to get Grok, an AI chatbot developed by Musk’s xAI, to produce sexual images of children, which is against the company’s user guidelines.
The company did not immediately respond to a request for comment. In a post on X, the Grok chatbot said that child sexual abuse material (CSAM) was “illegal and prohibited”.
The incident prompted French ministers on Friday to report the sexual images generated by Grok to prosecutors. They also referred it to the country’s media regulator, Arcom, regarding “a possible breach by X” of its obligations under the EU’s Digital Services Act.
“The ministers condemn these acts in the strongest possible terms and reiterate the government’s unwavering commitment to combating all forms of sexual and gender-based violence,” the Finance Ministry said.
There have been several chatbot glitches over the past year. In July, Grok repeatedly praised Adolf Hitler and shared anti-Semitic rhetoric.
The latest incident will raise further concerns about how easy it is to exceed safety guardrails in AI models. The tech industry and regulators are grappling with the far-reaching societal impacts of generative AI.
Generative AI has led to an explosion in AI-generated sexual images of children and non-consensual deepfake nude images, as freely available AI models with no content safeguards and “nudified” apps make it easier than ever to create illicit images.
Internet Watch FoundationA UK-based non-profit said AI-generated images of child sexual abuse material has doubled in the last year becoming more aggressive,
The Grok is intentionally designed to have lower material guardrails than competitors, with Musk calling the model “maximum truth-seeking.”
The Grok 4, XAI’s latest, most powerful model, was released in July, featuring a “Spicy Mode” feature that allows users to generate bizarre and sexually suggestive content for adults.
Musk’s two-year-old AI start-up acquired X in March in a $45 billion all-stock deal. The transaction valued the combined company at $113bn. X incorporates some XAI features, such as Grok, directly into the platform.
The laws governing harmful AI-generated content are complicated. In May 2025, the US signed the Take It Down Act, which tackles so-called AI-generated “revenge porn” and deepfakes.
The UK is also working on a bill that would make it illegal to possess, create or distribute AI tools that generate CSAM, and require AI systems to be thoroughly tested to check that they cannot generate illegal content.
In 2023, researchers at Stanford University discovered that a popular database used to build AI-image generators was filled with CSAM.
