Most policymakers are familiar with the basic approach to monitoring policy outputs (such as the number of people who have gone through a training programme, and how many got a job). I have done this hundreds of times in my career. However, in economic development we tend to be less practiced at isolating the causal impact of policy interventions.
By causal impact, the evaluation literature means the difference between the outcome for individuals ‘treated’ in a programme, and the outcome they might have experienced without it. Pinning down causality is a crucially important part of impact evaluation. After all,estimates of the benefits of a programme are of limited use to policy-makers unless those benefits can be attributed, with a reasonable degree of certainty, to that programme.
Establishing causality requires the construction of a valid counterfactual – i.e. what would have happened to programme ‘participants’ (individuals, firms or areas) had they not been treated under the programme. The way in which this counterfactual is constructed is the key element of impact evaluation design.
A standard approach is to create a counterfactual group of similar individuals, firms or areas not participating in the programme being evaluated. Changes in outcomes can then be compared between the ‘treatment group’ (those affected by the policy) and the ‘control group’ (those not affected by the policy).
Another approach – useful if we are interested in figuring out ‘what works better’ – is to offersimilar groups different treatments. For example, in the case of business support, we take two similar types of business and offer some mentoring support (more expensive) and others are referred to online materials (a less expensive type of advice). The two groups can then be tracked to see how the different approaches play out.
The fundamental idea here is to make sure we are comparing groups that are similar. Such comparisons help address concerns that other factors might be driving changes for participants in a programme. These other factors might be ‘external’ to the programme (for example, it is offered to struggling areas) or they might be ‘internal’ (e.g. only certain types of firms choose to take part in a business advice programme). There is a lot more detail which can be considered when choosing a control group – and our selection of case studies for particular policies provide examples of how evaluations have dealt with these issues in practice. But even for those with less technical expertise – thinking about suitable comparison groups is a crucial step in thinking how you might evaluate policy success, as well as what you can learn in the absence of a comparison.
If you find yourself convinced, but are struggling to convince partners, you may find Ben Goldacre’s succinct description on Newsnight in February provides an invaluable introduction. And of course, we’re always available to try to help those looking to undertake more robust evaluation.