Elon Musk’s xAI has restricted image generation on its Grok AI model after a wave of non-consensual sexual deepfakes — including images of minors — spread across X and the Grok app, prompting investigations and legal threats from governments worldwide.
What xAI changed
xAI said it had implemented technical measures to stop Grok from producing certain sexually explicit images. In a post on X, the company said Grok would no longer allow the editing of images of real people into revealing clothing such as bikinis, and that generating images of real people in bikinis, underwear or similar attire through Grok would be blocked in jurisdictions where it is illegal. The week before, xAI had limited use of the Grok image generator to paying customers — a step critics said risked monetising the problem rather than solving it.
The company also said it acts to remove high-priority offending content, including child sexual abuse material (CSAM) and non-consensual nudity, and takes action against accounts that violate its rules. Reporting at the time indicated the safeguards were incomplete, with some users still finding ways around them.
The scandal that triggered it
The changes followed public outcry after large numbers of people used Grok to create sexual deepfakes of women — and, in some cases, children — and shared them on X and the Grok app, both operated by xAI. The abuse drew condemnation from child-safety advocates and rapid attention from regulators in multiple countries.
The regulatory response
In the United Kingdom, the government said it would accelerate new powers making it a criminal offence to create or take non-consensual intimate images, including AI-generated ones. Prime Minister Sir Keir Starmer said officials had made clear to X that such images are illegal and must be addressed, and that the platform had indicated it was working to comply with the country’s online safety laws. The UK regulator Ofcom opened an investigation.
In the United States, California’s attorney general announced an investigation into Grok and xAI over undressed and sexual AI images of women and children. The European Commission said it was weighing further measures and signalled it would use the full enforcement toolbox of the Digital Services Act (DSA) if xAI’s changes proved ineffective. Grok has also faced access restrictions in countries including Indonesia and Malaysia, and Musk had earlier pushed back against UK threats to restrict the platform.
Limitations and what to watch
The situation was fast-moving, and several details continued to evolve. xAI’s restrictions applied unevenly across jurisdictions and, by multiple accounts, did not fully prevent abuse, so their real-world effectiveness remained an open question. Investigations by regulators in the UK, US, EU and elsewhere were at early stages, and outcomes — fines, mandated changes or litigation — had yet to be determined. Reports also pointed to turnover on xAI’s safety and security teams around this period, though the full picture was unclear. The episode is best read as one flashpoint in a broader, unresolved debate over how to govern generative-image tools and hold platforms accountable for non-consensual and illegal content.
This article concerns AI-generated abuse imagery and its regulation. Anyone affected by image-based abuse can seek help from national support organisations and law-enforcement channels in their country.
