Process Tracing: The Potential and Pitfalls for Impact Evaluation in International Development. Summary of a Workshop held on 7 May 2014

Abstract

In international development there is increasing pressure to demonstrate that aid spending is making a difference. In short, that it is having an ‘impact’. During the past decade, there has been a rapid rise in the use of experimental and quasi-experimental evaluation designs – viewed by some as the ‘gold standard’ in evaluation methodologies. Such designs are based on a counterfactual logic of assessing causation/attribution; an approach that requires large-n studies and quantitative datasets in order to test the statistical difference in outcomes between the treatment and a properly constructed comparison group (the control).

There is now, however, a growing interest in exploring alternative, yet still robust, approaches to impact evaluation. This is particularly so where the nature of the intervention or the context demands a small-n study (such as interventions that seek to influence policy through engaging civil society actors and other initiatives). It is in this context that process tracing offers much potential; as both an established social science research methodology and one that, at its core, focuses on investigating causal mechanisms.

The Institute of Development Studies (IDS) is engaged in a four-year programme of work entitled Strengthening Evidence-based Policy, funded via an Accountable Grant (AG) from the Policy Division of the UK Department for International Development (DFID). The grant contributes to improving the lives of the poorest by expanding evidence-based knowledge, policy options and guidance in six major thematic areas, and a seventh theme focusing on three cross-cutting issues. As part of this work, the AG will assess the impact of its ‘policy interventions’ to bring evidence to bear on policy change. For many reasons, these policy interventions are not amenable to large-n evaluation designs, and many across the Institute and elsewhere are considering the potential of process tracing (see, for example, Barnett and Befani 2014).