‘How to’ resource
Substantial disruptions to local economic activities cause local economic shocks. The most common type of shock is the closure (planned or unplanned) of a major local employer, such as Honda’s manufacturing plant in Swindon in 2021 and the SSI steelworks in Tees Valley in 2015.
Most areas experience major shocks infrequently and public sector responses often need to be developed quickly. These constraints can make formulating an effective response difficult. Having a pool of evidence on ‘what works’ can help national and local policymakers design their response. Building this evidence base requires impact evaluation on past responses to be undertaken and shared.
This briefing sets out guidance for local and national policymakers on evaluating their responses to local economic shocks.
Whilst the briefing has been written for major shocks (such as the complete closure of a significant local employer that will have impacts on employees, suppliers, and land use), many of the approaches could also be used for other shocks (such as a large reduction in headcount within a local employer). These relatively smaller shocks will generally have narrower impacts (for example, on employment but not land use, or on workers but not supply chain businesses). Evaluations of policy responses to shocks at different scales all add to the evidence base.
Key messages
Impact evaluation of responses to local economic shocks
Impact evaluation establishes whether an intervention leads to changes in outcomes. Few impact evaluations of public sector responses to local economic shocks have been undertaken. One reason for this is that responses to shocks often need to be developed quickly, which limits options. Evaluations are also often dependent on the co-operation of the business directly or indirectly affected by the shock.
Establish if impact evaluation is what is needed – other options such as monitoring or process evaluation may be more appropriate.
Engaging partners
Consider who will need to be involved. Responses to local economic shocks are likely to include a range of partners and evaluation will need to reflect this. Partners should be involved at an early stage to help ensure their buy-in to the evaluation and, where relevant, to act on its findings.
Establishing what to evaluate
Decide which elements of the response are most important to evaluate, including whether the objective is to understand whether the intervention delivered the intended outcomes (‘what works’) or which interventions or design features are most effective in achieving the outcomes (‘what works better’).
Use a logic model to understand what the intervention is seeking to achieve and how it will achieve this. Consider developing logic models for both the response as a whole and for each intervention. Identify a small number of specific outcomes to be evaluated.
Assessing the feasibility of impact evaluation
Impact evaluations establish causality by using comparison. The lack of an available comparator – another area that is going through a similar shock but that did not receive support – makes it highly unlikely that most policy responses to local economic shocks can be evaluated. Evaluation of individual elements of a response, for example, training to help redeploy redundant workers, or marketing of the site to attract new tenants, may be possible if a comparison group can be established.
Identifying a comparison group for specific interventions
Identify a comparison group that is similar to the treatment group. Key considerations include the factors that are likely to affect the outcome of interest and how beneficiaries will be selected into treatment.
Establishing data requirements and processes
Establish what data is needed and put in place processes for its collection and analysis. Data sharing will be needed in most cases. Having a pre-emptive data sharing agreement in place amongst partners that would be enacted in the event of a shock would be extremely useful.
Other considerations
Consider when to evaluate, recognising that the need for insights to inform policymaking often comes before outcomes have been realised. Previous evaluations can be used as inspiration for evaluation design.
Value for money evaluation
Value for money evaluation should focus on individual elements of response, drawing on data collected through monitoring and impact evaluation.
Note
This briefing has been developed in collaboration with Andy Pike, Henry Daysh Professor of Regional Development Studies, Centre for Urban and Regional Development Studies (CURDS), Newcastle University, UK, and is based on national and local roundtables with relevant policymakers and researchers. Andy authored the What Works Centre for Local Economic Growth evidence briefing on local responses to economic shocks.
