Overseas 'content farms' creating political deepfakes uncovered
Getty ImagesOverseas "content farms" are using Artificial Intelligence (AI) to create social media posts about UK politics, an expert has warned.
Technology company Meta removed several Vietnam-based pages from Facebook after a BBC Wales investigation found they were spreading fake news.
The warning, from Prof Martin Innes of Cardiff University, comes as the Electoral Commission is developing software to spot and combat deepfakes ahead of the Welsh and Scottish parliaments' elections in May.
The BBC has also uncovered examples of AI-generated videos, shared by pages in Wales, falsely showing Welsh politicians in compromising situations including endorsing a rival and kissing a colleague.
Several Welsh politicians told the BBC about their experiences as victims of deepfakes.
"I don't think you'll find a politician who hasn't had this done to them... to say it out loud makes me feel quite sad," said Labour MP Alex Davies-Jones.
What is a deepfake?
Deepfakes are videos, pictures or audio clips digitally altered or manipulated to make something fake look real.
Some are silly, but others could potentially damage reputations.
And text-to-image AI tools have made them easier to generate than ever before.
BBC Wales found a number of Facebook pages publishing fake news about UK politicians, often accompanied by AI-generated images.
Each of these pages had thousands of followers and many of them posted the same, or very similar, content.
They often had names which implied they were UK-based news outlets - but, by using a transparency feature on Facebook, it was possible to see almost all of these pages were run from Vietnam.
Although they featured some real news stories, a large amount of their content could be easily proven false.
Meta removed some of these pages after it was contacted by the BBC but, throughout the investigation, new pages were being created almost daily.
The pages often depicted multiple politicians, including Nigel Farage, Boris Johnson, Rishi Sunak and Zia Yusuf, in the same fake situation – for example, storming out of a BBC interview after dramatic arguments with Laura Kuenssberg.
There is no suggestion these pages are being run by, or on behalf of, any political party.
Prof Martin Innes, director of Cardiff University's Crime and Security Research Institute, said the pages were "content farms" designed to go viral.
They may in turn make their owners money from Facebook's monetisation programme, but it is not possible to know for sure whether each page is monetised.
Facebook marked some stories with a warning, saying they had been proven false by third-party fact-checkers - for example after Full Fact debunked a story about Reform UK leader Nigel Farage being in hospital.
But on each occasion it was possible to find very similar stories that did not have this warning attached.
Many of the stories posted by these pages appeared to be designed to get attention from supporters of Farage, but not all were positive about him.
In some he was falsely depicted adopting dogs, donating his personal wealth to good causes, or welcoming a new baby.
But another showed him being arrested, complete with AI-generated photos of him in handcuffs.
The pages also feature other politicians including Sir Keir Starmer, falsely saying he had been taken ill on stage, sued over "election rigging" and removed as Prime Minister.
It was not always politicians – a wide range of celebrities could be seen inserted into some of the same made-up situations too.
Innes said pages like these were "driven by a profit imperative," and would "do anything where they think they can get eyeballs on their content if it will make them money".
Some of these posts appeared to get lots of likes, comments and shares, while others had almost none despite the pages' high follower counts.
Innes said we could not be sure whether all of the pages' thousands of followers were real people, adding page owners often used bots to "spoof the algorithm" and get content into people's feeds.
The comments section suggested not everyone who saw this content believed it.
Many expressed doubt or annoyance at the fake stories – but others did seem to be taken in by it.
When the BBC approached Meta with the examples found, the company removed several of the pages from Facebook.
It added it had a policy against the use of "inauthentic" accounts or pages on its platforms.
How could deepfakes affect the Welsh election?
Devolved elections will take place in Wales and Scotland on 7 May, the same day as local elections in parts of England.
There have been warnings before of a so-called 'deepfake election' in the UK - and mixed opinions on whether this has ever materialised.
The Alan Turing Institute, a national centre for data science and AI, found "no evidence" that AI-enabled deepfakes or disinformation had a meaningful impact on the result of the general election in 2024.
But almost two years on, the "barriers to entry" to creating this kind of content has been lowered, said Innes.
"Where previously doing this kind of image [or] video manipulation would have required quite a lot of computing power and technical knowledge, you don't need that anymore."
This, he said, could create a "trickle-down effect" where videos start to impact devolved politics.
The risk is high enough for the Electoral Commission to get involved.
The independent body, which oversees and regulates UK elections, is working with the Home Office on software to identify, track and report deepfakes.
Its chief executive Vijay Rangarajan said it would "help voters identify misinformation during election campaigns and reduce activity which negatively impacts voters' confidence in the conduct of campaigners".
Following this announcement, the BBC identified a number of AI-generated fake videos of Welsh politicians, created in recent weeks.
Between them, the videos have more than 200,000 views.
These included a fake video of Prime Minister Keir Starmer and Wales' First Minister Eluned Morgan kissing, and another of Plaid Cymru leader Rhun ap Iorwerth shouting "I love Reform".
The owners of the Facebook page behind these two videos said they were "obvious" satire.
They added they trusted their audience to use "common sense" but indicated the videos were fake in their captions.
Another, shared by an anti-Reform page, showed Nigel Farage with a Welsh flag behind him.
In the video, he shouts: "Britain is great; I don't see the point of having four different sporting teams and four different parliaments."
There is no record of Farage ever saying these words, and the video appeared distorted in ways consistent with AI generation.
The page behind this video has been contacted for comment.
These particular examples, Innes said, were closer to "shallowfakes" than "deepfakes" - meaning they were less realistic and were created using less sophisticated software.
But identifying fakes was getting "harder and harder", he added, with his team sometimes spending a day "pouring over" the more sophisticated examples.
He said the Electoral Commission's plans to combat deepfakes "might help us understand things after the event" but would not stop or prevent deepfakes affecting the election.
Davies-Jones, who represents Pontypridd, Rhondda Cynon Taf, said she had been subjected to "horrific" deepfakes, including some showing her in "crude sexual positions" in underwear.
She added speaking out about non-consensual sexual deepfakes could be "embarrassing" and lead to "increased targeting" - and their existence was "no good for any healthy democracy".
Llŷr Powell, who was the Reform UK candidate in the Caerphilly by-election, said his team had found AI-generated videos of him and his colleagues which put words in his mouth, lying about his policies.
Powell said he was "in favour of free speech" and freedom of expression but there was a problem when AI use led to "some people believing that it was fact and not parody".
He added: "We don't want to sacrifice the freedoms we've got in this country, but at the same time the public have a right to know what information they're being fed and the tool is being misused here just like anything can be misused."
Janet Finch-Saunders, Conservative MS for Aberconwy in north Wales, said she "felt sick" when she saw a photo of her had been edited "in a very disgusting way" to create an explicit deepfake.
She said the fakes could be "very misleading" to people who were "less tech-savvy" and "it is quite worrying because there's far more capability now".
"When you're in politics... or whatever your line of work, you should not be attacked personally. And certainly not AI used to belittle you."
Baroness Carmen Smith, a Plaid Cymru peer, said: "Deepfakes seriously are a threat to democracy, but they are also a threat to many, many people whether that is your sister, your friend, your colleague.
"We have to keep up to date with what developments are being made with AI tools... it's something that we do need to address and prioritise now."
Rachel Millward, deputy leader of the Green Party of England and Wales, said she had been subject to a "misogynistic deepfake" herself and called on the UK government to regulate the AI sector to prevent such content "undermining the legitimacy of future elections".
A Welsh Liberal Democrat spokesperson said deepfakes were a "threat to the safety of politicians across the political spectrum" and called for "cross-nation action".
The UK government's department for science, innovation and technology said the "potential for deepfakes to sow division, spread false information, and influence public opinion" was "well recognised".
It said social media platforms must "proactively tackle illegal fraudulent content" under the Online Safety Act, or "face enforcement action".
BBC Wales Your Voice Live: Ask the Leader wants you to be in the audience and put your questions to the leaders of Wales' parties.
Click here to put your questions to the leaders of the Welsh Conservatives, Plaid Cymru and the Welsh Liberal Democrats in Haverfordwest on 8 April.
Click here to put your questions to the leaders of Welsh Labour, Reform UK and the Wales Green Party in Llandudno on 15 April.
