In November last year, Muralikrishnan Chinnadurai was watching a live broadcast of a Tamil-language event in the UK when he noticed something strange.
A woman introduced as Duwaraka, daughter of Velupillai Prabhakaran, the militant leader of the Tamil Tiger, was giving a speech.
The problem was that Duwaraka had died more than a decade ago, in an airstrike in 2009, during the final days of Sri Lanka’s civil war. The 23-year-old’s body was never found.
And now, here she was – apparently a middle-aged woman – urging Tamilians around the world to take up the political fight for their freedom.
Chinnadurai, a fact-checker in the southern Indian state of Tamil Nadu, watched the video carefully, noticed flaws in the video and quickly attributed it to an artificial intelligence (AI)-generated figure.
The potential problems were immediately clear to Mr. Chinnadurai: “This is an emotive issue in the state [Tamil Nadu] and with elections just around the corner, misinformation could spread quickly.”
As India goes to the polls, it’s impossible to avoid creating a wealth of AI-generated content – from campaign videos to personalized audio messages in a variety of Indian languages and even automated calls made to voters in a candidate’s voice.
Content creators like Shahid Sheikh have even had fun using AI tools to show Indian politicians in avatars we’ve never seen them in before: wearing sportswear, playing music and dancing.
But as the tools become more sophisticated, experts worry about their implications when it comes to making fake news seem real.
“Rumors have always been part of electoral propaganda. [But] In the age of social media, it can spread like wildfire,” says SY Qureshi, the country’s former chief election commissioner.
“This could really set the country on fire.”
India’s political parties are not the first in the world to take advantage of recent developments in AI. Just across the Pakistan border, he allowed jailed politician Imran Khan to attend a rally.
And in India itself, Prime Minister Narendra Modi has also made the most of emerging technology to run effective campaigns – addressing an audience in Hindi which, using the AI tool created by the Bhashini government, was then translated into Tamil in real time.
But it can also be used to manipulate words and messages.
Last month, two viral videos showed Bollywood stars Ranveer Singh and Aamir Khan campaigning for the opposition Congress party. Both filed complaints with the police saying that they were deepfakes, made without their consent.
Then, on April 29, Prime Minister Modi raised concerns about the use of AI to distort speeches by senior ruling party leaders, including himself.
The next day, police arrested two people, one from the opposition Aam Aadmi Party (AAP) and the Congress party, in connection with a doctored video of Home Minister Amit Shah.
Modi’s Bharatiya Janata Party (BJP) has also faced similar accusations from opposition leaders in the country.
The problem is that – despite the arrests – there is no comprehensive regulation, according to experts.
Which means that “if you get caught doing something wrong, at best you could get a slap in the face,” according to Srinivas Kodali, a data and security researcher.
In the absence of regulation, creators told the BBC they have to rely on personal ethics to decide what kind of work they choose to do or not do.
The BBC found that, among the politicians’ requests, were pornographic images and the transformation of videos and audios of their rivals to damage their reputation.
“I was once asked to make an original look like a deepfake because the original video, if shared widely, would make the politician look bad,” reveals Divyendra Singh Jadoun.
“So his team wanted me to create a deepfake that they could pass off as the original.”
Jadoun, founder of The Indian Deepfaker (TID), which created tools to help people use open-source AI software to create campaign material for Indian politicians, insists on putting disclaimers on everything he does so that it stays Of course it’s not real.
But it is still difficult to control.
Sheikh, who works for a marketing agency in the eastern state of West Bengal, has seen her work shared without permission or credit by politicians or political pages on social media.
“A politician used an image I created of Mr. Modi without context and without mentioning that it was created using AI,” he says.
And now it’s so easy to create a deepfake that anyone can do it.
“What used to take seven or eight days to create can now be done in three minutes,” explains Jadoun. “You just need to have a computer.”
In fact, the BBC has seen first-hand how easy it is to create a fake call between two people – in this case, me and the former US president.
Despite the risks, India initially said it was not considering an AI law. In March this year, however, it took action following a furore over Google’s Gemini chatbot’s response to a question: “Is Modi a fascist?”
Rajeev Chandrasekhar, the country’s junior information technology minister, said this violated the country’s IT laws.
Since then, the Indian government has asked technology companies to obtain its explicit permission before publicly releasing “untrusted” or “undertested” generative AI models or tools. He also warned against responses from these tools that “threaten the integrity of the electoral process”.
But it’s not enough: Fact-checkers say that keeping up with the debunking of such content is a difficult task, especially during elections when misinformation is at its peak.
“Information travels at the speed of 100 kilometers per hour,” says Chinnadurai, who runs a media watchdog in Tamil Nadu. “The debunked information we put out will go 20 miles per hour.”
And these fakes are even reaching mainstream media outlets, says Kodali. Despite this, the “electoral commission is publicly silent on AI”.
“There are no general rules,” says Kodali. “They are letting the tech industry regulate itself instead of creating real regulations.”
There is no foolproof solution in sight, experts say.
“But [for now] If action is taken against people forwarding false information, it could scare others against sharing unverified information,” says Qureshi.
