Regulators say the case could become a defining test of how far governments are willing to go in holding AI systems and the platforms that deploy them responsible for the most extreme forms of digital harm, at a moment when generative tools are spreading faster than the rules meant to contain them.
Elon Musk’s social media platform X is under investigation by regulators in Europe, India and Malaysia after its Grok chatbot was used to generate and circulate explicit, sexualized images of women and children, including material that authorities say may amount to child sexual abuse.
British media regulator Ofcom said it has requested information from X about the issue.
In Brazil, a member of Parliament said on social media that she had asked federal prosecutors and the national data protection authority to suspend Grok’s operation in the country until an investigation is completed.
The scrutiny follows a surge in recent weeks of Grok-generated nonconsensual intimate images, known as NCII, created from photographs or videos of real people in response to user prompts. Many of the images were widely shared on X, alarming child safety advocates and digital rights groups.
The problem emerged after Musk’s company updated Grok’s image generation features, making it easier for users to create images directly from text prompts within the platform.
While critics and safety experts condemned the spread of exploitative content, Musk appeared to play down the backlash.
He shared several Grok-generated images on X, including one portraying himself in a bikini, accompanied by laughing-crying emojis.
At a press conference, Thomas Regnier, a spokesperson for the European Commission, said regulators were taking the issue seriously and were aware that X and Grok were “now offering a spicy mode showing explicit sexual content with some output generated with childlike images.”
“This is not ‘spicy,’” Regnier said. “This is illegal. This is appalling. This is disgusting. This is how we see it, and this has no place in Europe.”
In India, the Ministry of Electronics and Information Technology last week ordered X to conduct a “comprehensive technical, procedural and governance-level review” of Grok, giving the company until Jan. 5 to comply. In Malaysia, the Communications and Multimedia Commission said it was investigating X and would summon company representatives.
“MCMC urges all platforms accessible in Malaysia to implement safeguards aligned with Malaysian laws and online safety standards, especially in relation to their AI-powered features, chatbots and image manipulation tools,” the commission said in a statement.
In the United States, the National Center on Sexual Exploitation called on the Department of Justice and the Federal Trade Commission to investigate.
Dani Pinter, the group’s chief legal officer and director of its law center, said there was “not a lot of legal precedence on point for these specific issues,” but noted that federal laws already prohibit the creation and distribution of child sexual abuse material, or CSAM.
Those laws, she said, can apply to virtually created content “when it depicts an identifiable child, or depicts a child engaged in sexually explicit conduct.” She pointed to statutes including the Take It Down Act, which was endorsed by First Lady Melania Trump before being enacted last year.
A Justice Department spokesperson said in a statement that “The Department of Justice takes AI-generated child sex abuse material extremely seriously and will aggressively prosecute any producer or possessor of CSAM.
We continue to explore ways to optimize enforcement in this space to protect children and hold accountable individuals who exploit technology to harm our most vulnerable.”
The Federal Trade Commission declined to comment. XAI, the artificial intelligence company behind Grok, did not provide a response beyond an automated reply.
X addressed the controversy publicly for the first time on Saturday in a post from its official safety account.
“We take action against illegal content on X, including Child Sexual Abuse Material (CSAM), by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary,” the company wrote.
Musk added in a separate post, “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content.”
The following day, an xAI employee, Ethan He, said in a post that Grok Imagine had been updated, though he did not specify whether the changes addressed the ability to create explicit or harmful images.
Musk and X have previously faced criticism for their handling of child sexual exploitation content.
In 2023, the platform briefly suspended, then reinstated, the account of Dom Lucre, a right-wing influencer who had posted images linked to the criminal conviction of an Australian man in the Philippines, according to Mashable.
Musk said at the time that the posts would be deleted but the account reinstated. Lucre’s account remains monetized and lists 1.6 million followers.
Tom Quisel, the chief executive of Musubi AI, a company that helps social networks automate content moderation, said xAI appeared to have failed to implement even basic safeguards when rolling out Grok’s image tools.
“It would be easy for a company like xAI to have its model detect and block an image involving children or partial nudity,” Quisel said, or to reject prompts that place people in sexually suggestive scenarios.
Despite the controversy, the episode appears to have had little negative impact on X’s popularity.
Data from Apptopia shows daily downloads of the Grok app have risen 54 percent since Jan. 2, while daily downloads of X increased 25 percent over the past three days.
Regulators across several continents are now weighing whether existing laws are sufficient to curb the misuse of powerful generative AI tools, or whether new rules are needed to rein in platforms that allow such content to proliferate.
Stay ahead in the world of AI, business, and technology by visiting Impact AI News for the latest news and insights that drive global change.
Got a story to share? Pitch it to us at info@impactnews-wire.com and reach the right audience worldwide!
Discover more from Impact AI News
Subscribe to get the latest posts sent to your email.
