Do you ever see comments on social media that seem way off topic, but still manage to wrench the discussion around to divisive political debate?
A discussion about the cost of living suddenly becomes an argument about immigration. A conversation about the war in Ukraine turns into claims about government corruption. It can feel jarring – and sometimes this is deliberate.
As generative AI becomes more powerful, malicious groups are increasingly using it to produce and spread disinformation online. Automated accounts can flood social media with convincing comments designed to sow division, inflame political debate and undermine trust in reliable information.
But our latest research offers a way to spot these attempts. Rather than trying to identify whether a post was written by AI, we focus on something different: whether it’s trying to derail the conversation.
Until recently, identifying malicious accounts was often quite straightforward. Many campaigns relied on people writing in a second language. So, posts sometimes contained grammatical mistakes or unusual word choices. Detection systems could look for these patterns in the language used.
But generative AI has changed that. AI systems can now produce fluent, natural-sounding text that is much harder to distinguish from human writing. For example, patterns like use of em-dashes and the word “delve” used to be telltale signs of a text being generated by AI. But AIs are adapting, and these older systems are increasingly ineffective.
Trying to detect AI purely from the words people use is becoming a losing battle. We believe the better approach is to look at what a message is trying to achieve.
Looking for signs
Attempts to spread disinformation often work by steering conversations away from their original topic, towards more polarising issues. So, instead of analysing individual words, we set out to build a system that could recognise this phenomenon in online discussions.
We analysed comments posted beneath BBC News videos on YouTube, a platform that has previously been targeted by organised disinformation campaigns.
For example, imagine a comment about Ukraine’s president, Volodymyr Zelensky, interacting with senior UK political figures: “Zelensky must be wondering how many foreign secretaries the UK goes through.” Now, imagine another person responding: “Mind you, Zelensky has barely been president for four years. Maybe that’s why the little tyrant bans his opposition.”
Whether that second point is true or false is not the issue. Instead of responding to the original comment, it redirects the conversation towards a different, more divisive topic.
This is known as a red herring: introducing an unrelated issue that distracts from the original discussion. These kinds of shift are difficult for conventional disinformation detection systems to identify, because they are not tied to particular words or phrases.
We manually analysed more than 1,600 comments under BBC News videos, labelling them according to 25 different features of online discussion – and discovered some clear patterns.
We found that 36% of derailing messages had red herrings, 65% had leaps in logic known as “non sequiturs”, and 20% contained personal attacks. They were also much less likely to acknowledge previous comments or express empathy.
Spotting manipulation
The next step was to see whether an AI system could recognise these patterns automatically. We used an AI to catch an AI.
For every genuine online comment, we asked an AI large language model to generate several reasonable, relevant responses. Returning to the example of UK foreign secretaries, the AI suggested replies such as: “The current situation in this country must come as quite a shock” or “One too many?”. Both responded directly to the original point.
The system then compares the real response with our AI-generated replies. If the actual comment differs substantially, it may indicate that someone is attempting to steer the conversation in a different direction. So, rather than searching for suspicious words, our system looks for unexpected changes in the flow of the discussion.
How the system works:
We tested this approach using our manually labelled dataset. In our second study, the system correctly identified derailing comments around 77% of the time.
That’s far from perfect, but no detection system is – particularly when analysing something as complex as human conversation. However, our approach performed around twice as well as existing systems based on word-level sentiment analysis. It also achieved results comparable with the level of agreement between human researchers.
Our approach is effective because the AI learns what a typical response to a conversation looks like. When a reply unexpectedly changes the discussion, the system can identify that change and analyse patterns that earlier methods couldn’t detect.
Read more: Why science GCSEs matter more than we think in a post-truth age
Of course, going off topic isn’t necessarily a sign of malicious intent or disinformation. People naturally take conversations in unexpected directions, and there are many legitimate reasons why discussions evolve.
For that reason, this technology may act as an early-warning system rather than a replacement for human judgment. It could help moderators identify conversations that deserve closer attention – but any final decisions should remain with trained experts.
There are also important ethical questions to address. AI systems can reflect biases in the data they are trained on, and they still do not understand conversations in quite the same way that people do. Improving how AI represents and interprets human discussion remains a challenge.
As AI-generated content becomes increasingly difficult to distinguish from human writing, detecting disinformation requires more than simply searching for telltale words. It requires understanding how conversations work, how they are manipulated, and when someone is trying to quietly steer them off course.
Comments are closed.