OpenAI says Russian and Israeli groups used its tools to spread disinformation

OpenAI on Thursday released its first report on how its artificial intelligence tools are being used for covert influence operations, revealing that the company has disrupted disinformation campaigns originating in Russia, China, Israel and Iran.

Malicious actors used the company’s generative AI models to create and publish advertising content on social media platforms and to translate their content into different languages. Neither campaign gained traction or reached large audiences, According to the report.

Related: US$10 million prize launched for team that can actually talk to animals

As generative AI has become a booming industry, there has been widespread concern among researchers and policymakers about its potential to increase the quantity and quality of online misinformation. Artificial intelligence companies like OpenAI, which makes ChatGPT, have tried, with mixed results, to assuage these concerns and put barriers in their technology.

OpenAI’s 39-page report is one of the most detailed accounts from an artificial intelligence company on the use of its software for advertising. OpenAI said its researchers found and banned accounts associated with five covert influence operations over the past three months, coming from a mix of state and private actors.

In Russia, two operations created and disseminated content criticizing the US, Ukraine and several Baltic countries. One of the operations used an OpenAI model to debug code and create a bot that posted to Telegram. China’s influence operation generated texts in English, Chinese, Japanese and Korean, which the agents published on Twitter and Medium.

Iranian actors generated full articles attacking the US and Israel, which they translated into English and French. An Israeli political company called Stoic ran a network of fake social media accounts that created a range of content, including posts accusing US student protests against Israel’s war in Gaza of being anti-Semitic.

Several of the misinformation spreaders that OpenAI banned from its platform were already known to researchers and authorities. The US treasury sanctioned two Russian men in March, who were allegedly behind one of the campaigns detected by OpenAI, while Meta also banned Stoic from its platform this year for violating its policies.

The report also highlights how generative AI is being incorporated into disinformation campaigns as a way to improve certain aspects of content generation, such as creating more convincing foreign language posts, but that it is not the only propaganda tool.

“All of these operations used AI to some extent, but none used it exclusively,” the report stated. “Instead, the AI-generated material was just one of many types of content posted, along with more traditional formats such as handwritten text or memes copied from the Internet.”

While neither campaign resulted in a notable impact, the use of the technology shows how bad actors are discovering that generative AI allows them to increase propaganda output. Writing, translating and publishing content can now be done more efficiently through the use of AI tools, reducing the level of creation of disinformation campaigns.

Over the past year, malicious actors have used generative AI in countries around the world to try to influence politics and public opinion. Deepfake audio, AI-generated images, and text-based campaigns have all been employed to disrupt election campaigns, leading to increasing pressure on companies like OpenAI to restrict the use of their tools.

OpenAI said it plans to periodically release similar reports on covert influence operations, as well as remove accounts that violate its policies.