The strengths and weaknesses of automated fact-checking tools

What is automated fact-checking?

Given the speed with which mis- and disinformation spreads digitally, it is all the more important that fact-checkers have tools to help them analyze data flows and respond quickly to misleading claims when they appear. Automation can help them with different stages of the fact-checking process, but it is not yet in a position to handle the process fully from start to finish. 

Identifying misleading claims is the first step towards correcting them. That often begins with monitoring broadcasts, online news, and social media. For live political events, fact-checkers can deploy text-to-speech software (as the Squash system developed by Duke Reporters’ Lab does) or grab television subtitle feeds (as Full Fact’s Live system does) in order to get audio information into a text format more easily processed by machines. Then, natural language processing (NLP) software steps in to pick the text apart, looking for the kinds of words and phrases that usually express claims that might be tested. ClaimBuster is one piece of software that “marks up” sentences to highlight claims and skip over irrelevant statements. 

Verifying claims is a laborious process that involves researching, interpreting, and assessing information. Current AFC technologies are poorly suited to such tasks, except when it comes to straightforward information lookup (such as confirming the correct population size of a country). However, automation can save fact-checkers work by helping them filter out claims that they or their colleagues have already examined by checking the output of monitoring systems against databases of previous research. Software such as ClaimReview allows for easy tagging of fact-checks and of misinformation, simplifying the creation of collaborative, international databases. Full Fact has developed its Trends software to log where and when specific misleading claims appear, making it possible for fact-checkers to identify persistent sources and channels of mis- and disinformation.

Automation can also help with the dissemination of fact-checks. Chequeabot, for example, not only handles many of the tasks above, but it can also prepare social media posts for an editor at the Argentinian fact-checking organization Chequeado to review and then send out. Duke Reporters’ Lab has produced FactStream, which shows a pop-up fact-check summary live and on-screen during political speeches and debates whenever the Squash system identifies a claim in its fact-check database. 

What kinds of misinformation does this approach most effectively address?

Automated technologies are most advanced when it comes to monitoring political events and identifying mis- and disinformation on social media. 

What are the advantages of this approach?

Automation speeds up the work of professional fact-checkers. This is especially important in dealing with the massive and high-speed flow of information on the internet. 

What are some of the challenges for this approach?

Automation is a useful tool in the hands of fact-checkers, but it also has significant limits in terms of what it can achieve by itself. 

AFC depends on natural language processing technology, which tends to understand certain formats better than others (such as a political speech vs. a talk show). NLP can reliably spot straightforward, declarative statements, but it is less capable of identifying complicated claims or subtler forms of misleading information. NLP is more advanced for English than most other languages. Chequeabot is an important exception (for Spanish-language fact-checking), but no such tools are available for fact-checkers where local languages do not align with global markets or where information circulates in multiple, regional languages. 

Automation does not yet work well when it comes to verifying new and unchecked claims, in part because AFC requires authoritative data that is often unavailable. This is especially a problem where access to information is already limited, either because it is deliberately restricted by authoritarian regimes or simply because governments lack the resources to collect it. However, even where data is open and accessible, it also needs to be structured in ways that computers can easily process. According to communications scholar Lucas Graves, the result is that “fully automatic verification today remains limited to experiments focused on a very narrow universe of mostly statistical claims.”

How does it compare to other approaches?

Automation is a complement to other approaches but not a solution on its own. Investigative journalists are often in a better position to verify complex claims. 

What are some examples of best practice?

• Duke Reporters’ Lab, Tech & Check cooperative; 
• Full Fact Live and Trends; 
• Chequeado’s Chequeabot. 

Nuts and Bolts:

• Costs:    high for independent research and development; low for use of open-source software and tools shared by other organizations; 
• Funding:    grants from Google News Initiative, Facebook, etc.; pooling of resources among fact-checkers
• Topics:    mostly politics, economy, health;
• Inputs:    online news, social media, political events; 
• Outputs:    social media posts by fact-checkers; apps like Share the Facts.