Insights
Rigorous evaluation improves our understanding of the effectiveness of local economic policies. Robust evaluation can be challenging but can answer many questions and lead to a more nuanced understanding. By learning from impact evaluation, policymakers can better tailor projects to needs. Unfortunately, in the UK, we don’t do enough robust impact evaluation of local growth policies. At What Works Growth we want to help change that.
In our recent briefing, we outline lessons and recommendations for the evaluation of local growth policies drawing on a number of evaluations undertaken by What Works Growth over the years. These cover the Eat Out to Help Out scheme, Enterprise Zones, the Growth Vouchers Programme, and Local Major transport schemes.
Key recommendations for impact evaluation
General recommendations on impact evaluation are difficult to formulate as suitable designs can be context specific. However, there are some common lessons from our evaluations that we hope can make the process easier for others.
- Collect detailed information on the policy. Ideally, impact evaluation should happen early. Embedding it within project development can maximise options and reduce costs. If this is not possible, allocate time during planning to gather detailed information on timing, objectives, and the target population. One of the most challenging aspects of impact evaluation is selecting the right comparison group. Thoroughly understanding the policy and how the treatment was allocated, makes this task – and evaluation – easier.
- Choose your method carefully. If randomised controlled trials (RCTs) are not feasible, try to use a quasi-experimental design. Carefully adapt methods to the context and ensure that you consider the assumptions, data requirements, and limitations of each method, as these can affect the reliability and accuracy of results. Take advantage of the continual development of impact evaluation methods.
- Consider displacement effects. Spatially targeted policies carry the risk of local displacement, where economic activity relocates in an area without any other changes in response to the policy. This means that the overall impact on the local economy will be limited. In our evaluations, we show how using rings around intervention areas to assess displacement effects is a straightforward way to account for displacement in local growth policies.
- Define the relevant level of spatial units. While granular data is better for precision, aggregate data can be useful for a more macro perspective. Defining the correct spatial level will guide data needs for outcomes, treatment, and comparison groups.
- Integrate geospatial analysis. This approach enhances the analysis of effectiveness and makes it easier to identify to what extent policy impacts are localised. Assessing the spatial distribution of effects is useful feedback for delivery and policy colleagues locally and can help determine whether interventions target the appropriate areas.
- Consider the time and cost of data collection. Detailed design and planning are crucial for producing reliable results, but evaluations that are too expensive or time-consuming may be difficult to replicate or scale in other contexts or adapt to other growth policies. Balancing these factors helps to maintain quality while preventing delays that could render the findings less relevant for policymaking.
Why are these recommendations important?
Embarking on an impact evaluation can be daunting. Seeing examples of evaluations of UK policies can support analysts. These evaluations illustrate challenges and how these can be overcome to achieve a robust impact evaluation. Our hope is that with this briefing, analysts will have information that, combined with their theoretical and contextual knowledge, will help them implement a reasonable design in their own projects more quickly and smoothly. For more on the six recommendations, see our publication Lessons and recommendations from What Works Growth’s evaluations of local growth policies.
