‘How to’ resource
This checklist is intended to help policymakers to quickly assess the feasibility of undertaking an impact evaluation (i.e., experimental or quasi-experimental design with a counterfactual) for a particular intervention. An ‘intervention’ may be a project within a programme, a stand-alone project, or one element within a project.
With only five questions, this checklist is the first step in assessing evaluability. It can help to rapidly assess which interventions are clearly unsuitable for impact evaluation, leaving a narrowed list of projects for additional work on suitability.
Question: Strategic fit
Good evaluations start with a good understanding of how a policy, programme, or project fits into the wider policy context. The first question is about strategic fit and can help prioritise resources. If this answer is a ‘no’, other projects may be a better use of evaluation resource. If ‘yes’, but an answer below is a ‘no’, there may be ways that the intervention could be adapted for robust impact evaluation.
Question: Discrete
Impact evaluation establishes that the outcomes observed are due to the intervention by using a counterfactual – a comparison group that is similar to the treated group but didn’t receive treatment.
This requires being able to disentangle the outcomes expected from one intervention from others. If a programme is complex, with multiple activities with overlapping outcomes, measuring the change from one intervention may not be possible. We have a separate briefing that provides more information on impact evaluation at programme-level.
Question: Outcomes
To understand the impact of an intervention, a change in expected outcomes must be measurable. This requires a change large enough to be observable in the data. Marginal changes may be difficult to detect.
Question: Scale
Related to the magnitude of change in the outcomes is the scale of the intervention. The project needs to have enough participants so that the change measured has sufficient statistical power. A short note examining the scale of interventions covered in our evidence reviews will be published soon.
Question: Data availability
Evaluations rely on data. Data is needed on outcomes for both the treated and comparison groups, for before and after treatment. Data is also needed on the characteristics of the treated and comparison group. This question poses a question on data availability.
Next steps
This rapid assessment will narrow the number of interventions within a programme or pillar that should be explored further for impact evaluation suitability. A template to support the next step will be published soon. In the interim, our Transport evaluation proforma, although designed for transport interventions, provides a useful structure for discussions about evaluation for all interventions.
People within local government and central government department and agencies who are considering impact evaluation of local growth interventions can also contact us for advice and support on evaluation.
