Tracking success

Investigative journalism today needs to find new ways to prove its case. This may seem strange, as this special form of journalism has been extraordinarily successful in recent years. In the wake of the Panama Papers, for example, the world has seen penalties for individuals and companies in many countries as well as cases of concrete reform. 

"For every dollar spent on investigative reporting, hundreds of dollars are gained for societies from changes in public policy," writes James T. Hamilton in his in-depth study into investigative journalism,Democracy’s Detectives.

But this is not enough to ensure the survival of investigative journalism. The business model with which it has traditionally been financed is collapsing. Newspapers and other legacy media have been losing their advertising revenue and thus are less and less able to pay for in-depth reporting. 

Convinced of investigative journalism's positive repercussions for governance and society, different donors are willing to jump in and actively support such reporting. But in line with their funding principles, they wish to be informed about what has been achieved with the resources they made available. Meeting donor requirements is often a challenge for investigative reporters and their colleagues. How can they prove the impact of their reporting ?

What is impact?

To complicate things, the term "impact" has no consistent definition in the media context. Many media experts use it as a general term for the different kinds of effects reporting may have on audiences and societies. When it comes to investigative journalism, some experts suggest defining impact by differentiating between three different kinds of results. 

1. deliberative results: official commitments to discuss policy problems and their solutions
2. individualistic results: sanctions by policy makers against particular persons or enterprises
3. substantive reform: tangible regulatory, legislative, or administrative changes

In evaluation as well as in international development, "impact" is defined in a more technical, precise manner: It is understood as the highest level of change that can be achieved through an intervention. 

How can impact be measured?

According to theOECD, the (positive or negative) change effected by an intervention may be termed impact, if it is significant, extensive and transformative in nature. An impact can be achieved at the end of a result-chain which is based on "outputs" (products, capital goods and services), which may lead to "outcomes" (short-term and medium-term change and effects of intervention outputs) and which finally may make impact possible.

While different forms of outcome are achieved more often, impact in the stricter sense is rare in investigative journalism. And even less common is convincing evidence that lasting change is a direct result of a particular investigative report. 

However, the media may be more influential overall when it comes to molding the attitude and behavior of political elites, and through its muckraking it also acts as a deterrent. Investigative reports can also lead to important discussions on the values and standards a society wants to live by, laying the foundations for more fundamental change. 

Substantiating success

Different approaches are used to assess outcomes or impacts of investigative journalism. They tackle the problem from different angles, on different levels, and with different methodologies. In the following, key approaches are briefly described.  

As a first step, investigative reporters can use digital analytics tools to see how their stories are viewed, shared and commented on their websites and social media platforms. On top of this, it may then be useful to conduct agenda setting research by analyzing the way topics are presented and picked up by the media. 

This then includes a study of changes in framing amongst the public with the use of modern technology. For example, MIT Center for Civic Media's "Media Cloud" assesses how a story or a topic spreads across the web and social media. The UN Pulse Lab uses artificial intelligence to analyze the content of call-in radio shows to track how public discourse evolves. 

In order to manually document outcomes and impacts achieved through investigative reporting, different forms of tracking reports have been developed by non-profit organizations conducting investigative reporting. Whether this is done with a modern online tool or a simple spreadsheet is less important - what counts is that the information is gathered on different levels of outcome and impact. 

ProPublica, a foundation funded, non-profit investigative media outlet, assembles an internal "tracking report", which documents information according to the following categories: "action influenced by the story", "opportunities to influence change", "change that has resulted" and "lessons learned". In some cases, this monitoring is continued for many years – as long as the organization continues to report on the particular issue. 

Other examples that serve a similar function but also add various forms of tagging, categorization, and multi-use charting are the "Impact Tracker" developed by the Center of Investigative Reporting (CIR), Newslynx by the Tow Center for Digital Journalism at Columbia Journalism School and MORI (Measures of Our Reporting's Influence), an open source WordPress plug-in produced by Chalkbeat. These tools combine offers for the manual documentation of outcomes by the investigative team with some of the digital metrics described above. 

What affects the public agenda?

After completion of extraordinary projects, narrative accounts on the successes of investigative reporting are often produced. Here, investigative reporters or their colleagues tell the story behind the story of their muckraking. ProPublica, for example, publishes an annual impact report to provide an overview of its successes in achieving outcomes and impact.  

On top of this, an evaluation of change processes by independent experts can be of use. Some basic verification is performed by the jury-members of investigative reporting awards. An independent evaluation expert can go even further than simply checking the plausibility of the reporter’s accounts. Based on the documentation and interviews with different sources, an evaluator can conduct his own research and provide a new level of understanding of what was achieved and how. An example is the analysisof selected cases of African investigative journalism.  

Finally, academic research offers study designs to gauge the impact of investigative journalism, such as studying coalition building between different actors before and after investigative media reports. Two prominent and convincing examples of this kind of work as well as investigative journalism in general are the books "Journalism of Outrage" by David Protess and "Democracy's Detectives" by James T. Hamilton.  

 Conclusions

• Measuring outcomes and impacts of investigative journalism is complex, and it is not a digital enterprise only. What is needed is a set of flexible methods that can be combined and adapted to different situations. The evidence for success cannot be expressed in numbers or digital measurements alone. Instead, plausible arguments need to be presented in a narrative form. Still, some doubts and open questions may remain regarding the verification methods, models tested or the sequence of the results chains that led to the change. Overall, these measurements will only explain a limited part of investigative journalism and never fully assess all aspects of the importance of investigative journalism for democracy. 
• Measuring outcomes and impacts of investigative journalism can quickly become too resource intensive. Often a thorough assessment takes more time and funding than is available for newsrooms, award juries or donors. All involved must understand the limitations of the different approaches.
• Measuring outcomes and impacts of investigative journalism will play an increasingly important role in the future. While traditional investigative reporters are often reluctant to accept this, new actors from non-profit journalism and civil society are seeking new sources of funding and embracing and advancing new approaches. This does not mean that they do not differentiate between journalism and activism. Rather they are introducing innovative tools and work processes into investigative journalism practice. 
• A wise strategic choice on what can be realistically achieved in a given newsroom and situation is needed. For example, some basic documentation of the work done by the reporters, in combination with selected additional information, can already deliver a lot of insight. Also, using a Theory of Change approach that is flexibly adapted to changes in a given situation can help to motivate a team and give orientation. 
• New systems should be designed to help investigative reporters rather than to place additional burdens on them. Investigative teams should be encouraged to decide for themselves which tools and processes may be of use to them. These should be established in such a way that the time investigative reporters spend with documentation and filling application forms is limited to a minimum. Donors should also encourage honest tracking, learning and changes in the approaches. Failures and limited low-level outcome results are, after all, an integral and important part of investigative journalism.  

Jan Lublinski is Head of DW Akademie's Policy and Learning department

Literature

Coronel, Sheila (2008): Corruption and the watchdog role of the media. In: The News Media and the Governance Agenda. World Bank/Shorenstein Center. Conference Paper. Link

Graves, Lucas / Shabbir, Nabeelah (2019): Gauging the Global Impacts of the ‘Panama Papers' Three Years Later. Reuters Institute, University of Oxford. Link

Green, Elizabeth / Cramer, Philissa / Anand, Anika (2014): What We Talk About When We Talk About Impact. Chalkbeat white paper. Link

GIJN (2014): Investigative Impact. The Role of Investigative Journalism in Fostering Change and How to Measure it. Link

Hamilton, James T. (2016): Democracy's Dectectives. The economics of investigative journalism. Link

Jurrat, Nadine (2017): Investigative Journalism. With a little help from my audience. Link

Jurrat, Nadine / Lublinski, Jan / Mong, Attila (2018): Raking Muck and Raising Funds – Capacity Development Strategies for the Future of Investigative Journalism in the Global South. In: Investigative Journalism Education Consortium. Link

Kaplan, David E. (2013): Global Investigative Journalism: Strategies for Support. A Report for the Center for International Media Assistance. 2nd edition. Link

Learning for Action (2013): Deepening Engagement for Lasting Impact. A Framework for Measuring Media Performance & Results,” published by the Gates and Knight Foundations. Link

Lublinski, Jan / Spurk, Christoph / Fleury, Jean-Marc / Labassi, Olfa / Mbarga, Gervais / Nicolas, Marie Lou / Abou Rizk, Tilda (2016): Triggering Change - How investigative journalists in Sub-Saharan Africa contribute to solving problems in society. In: Journalism. Theory, Practice and Criticism. 17(8) 1074-1094. Link
Napoli, Philip M. (2014): Measuring Media Impact. An Overview of the Field. Link

OECD/DAC (2019): Better Criteria for Better Evaluation. Revised Evaluation Criteria. Definitions and Principles for Use. OECD/DAC Network on Development Evaluation. Link

Protess, David L / Cook, Fay Lomax / Doppelt, Jack C. et al. (1991): The Journalism of Outrage. Investigative Reporting and Agenda Building in America.

Schiffrin, Anya (ed., 2014). Global Muckraking. 100 Years of investigative journalism around the world.  

Schiffrin Anya (2014). Measuring Media Impact. Taking the long term view, supporting media independence. Link

Schiffrin, Anya / Lugalambi, George William (eds., 2017). African Muckraking.  

Schiffrin, Anya / Zuckerman, Ethan (2015): Can We Measure Media Impact? Surveying the Field. In: Standford Social innovation Review. Link

Stetka, Vaclav / Örnebring, Henrik (2013): Investigative Journalism in Central and Eastern Europe: Autonomy, Business Models, and Democratic Roles. In: The International Journal of Press/Politics 18 (4) 4113-435.

Tofel, Richard J. (2013) Non-profit journalism. Issues around impact. A whitepaper from ProPublica. Link

UN (2015): UN Global Pulse Lab. Analysing radio content. Link

Young, Nany / Suarez, Cyndi (2017): Unintended consequences. Should We Measure the Impact of Investigative Journalism. In: Columbia Journalism Review July 19. Link

Zhong, Nicole Zhaoqui (2018): Making impact a Key Performance Indicator. Link