The social media platform X has caused widespread backlash after users utilized its AI chatbot Grok to manipulate photos of females by removing their garments.
Several cases have surfaced where the technology has been employed to virtually strip women to render them in bikinis non-consensually, as well as to put them in inappropriate scenarios.
Developer Response and Regulatory Stance
XAI, the organization behind Grok, did not respond to requests for comment, except for an automatically-generated message stating "traditional news outlets lies".
A government spokesperson stated that legislation is being introduced to prohibit nudification software. Under a proposed criminal offence, anyone who supplied such technology would "be subject to a jail term and heavy penalties".
"The platform has the ability to stop these forms of misuse if they desired to," said a law professor, adding they "seem to have a lack of consequences."
Regulator Position and Tool Functionality
The regulator Ofcom said platforms must "assess the risk" of people accessing illegal content on their platforms, but stopped short of confirming whether it was currently investigating X or Grok in relation to AI-generated pictures.
Grok is a free AI assistant—with some premium features—which responds to X users' queries when they reference it in a post.
It is often used to provide commentary or additional context to other users' remarks, but people on X are also capable of modify an uploaded photo through its AI image editing function.
Background of Criticism and Policy
The system has been faulted for enabling users to produce pictures and clips with nudity and sexualised content, and it was previously alleged to have been involved in creating sexually explicit content of celebrities.
"X has been permitting the generation and distribution of these pictures for a long time without taking any action and we have not seen any intervention by regulators," Professor McGlynn added.
XAI's own terms of service prohibits "showing likenesses of persons in a graphic manner".
In a comment, Ofcom said it was against the law to "generate or distribute deepfakes or child sexual abuse material" and confirmed this applied to sexual deepfakes generated with AI.
It said services such as X were required to take "suitable steps" to "lessen the danger" of users encountering illegal material on their services and to delete it swiftly when they become aware of it.