Insights
In their focus on delivering a good policy on time and on budget, people often don’t think about evaluation until the end of a project (if at all). They then have to scramble around looking for ‘evidence’ to demonstrate its impact. Unfortunately, by this time it’s usually too late to provide convincing evidence of impact because the design of the scheme doesn’t allow for meaningful impact evaluation, or because basic data on the scheme (e.g. who got money) and how it operated (e.g. how decisions were made) are not available. In contrast, good evaluation is embedded in the policy design process. Sometimes this does happen in the UK (see, e.g. some of the high quality evaluations discussed in the NAO report on evaluation in government) but not for most local economic growth policies.
In fact, a large portion of the evidence base we have found on local economic growth policies comprises studies which look retrospectively at projects that have finished. Many of these studies aren’t even official evaluations – instead being undertaken by academics long after the policy has been implemented (and possibly completed). Evaluations after-the-fact can be useful – as our evidence reviews demonstrate – but there are many benefits to be gained from considering how a project will be evaluated from the earliest phases of project design. For example:
Deciding how the scheme would be evaluated means thinking about what data will provide the most accurate picture of the impact of the project. In turn, thinking about what data would capture success (or failure) focuses minds on which objectives are the highest priorities. This can help prevent the tendency to attempt to address too many diffuse goals with one policy. For example, when thinking about evaluating an SME support programme we might think about collecting data on employment or firm productivity. Which is better would depend on whether the policy was trying first and foremost to improve profitability, or employment, or productivity. Asking what is the most robust method to evaluate progress against each of these objectives helps decide what is the main objective.
Thinking early about good evaluation approaches (such as RCTs or quasi-experimental approaches), can help design the programme and the evaluation in a way that provides more insight into how a policy works. In other words, this can allow us to evaluate what works better (i.e. how to improve effectiveness) rather than just focusing on what works (i.e. whether the programme had any effect). For example, in this Swiss employment support programme, vouchers for different monetary values were randomly assigned to businesses entering a programme. This design feature allowed the policy to be assessed not only in terms of overall impact, but also value for money. Policymakers were able to assess whether there were thresholds above which offering more money produced diminishing returns – knowledge which should allow for improved cost-effectiveness as the programme is developed. Importantly, everyone in this programme also received some assistance: clever design got round the common objection that randomisation leaves some people without help. Thinking about these issues carefully can also help improve policy processes – because it requires precision on exactly how decisions will be made about who gets what support.
Early clarity about how the project’s impact will be measured also helps with the evaluation itself. Firstly, it allows for the use of more robust impact evaluation methods, while taking in to account other constraints on the policy design process. Second, it helps to ensure that the proper data is collected throughout the project life. Arrangements may need to be put in place to track participants over time, or count visitor numbers, or measure changes in profitability. This data will be more cheaply and accurately collected if good evaluation is built into the programme design, rather than if it has to be reconstructed later. It also allows data requirements for evaluation to be considered alongside those for monitoring – reducing duplication of time and effort if the two requirements are considered separately.
Finally, embedding evaluation in the design process forces us to think about how the results of the evaluation will feed back into future decisions about the programme. In the extreme, this might involve decisions about whether to continue funding the programme. Alternatively, evidence on what works better (e.g. the appropriate level of subsidy) can help us improve cost-effectiveness as a scheme is developed. Evaluations long after the programme is finished don’t help with these two crucial decisions – and undermine the most important way in which evaluations may directly improve policy effectiveness for the specific policy being evaluated. Evidence reviews are useful, but certainly second best compared to a feedback loop direct from evaluation of a specific policy to the future development of that policy.
There are other advantages to embedding evaluation in the design process, but these are four of the big ones. The crucial message that emerges is to think about how and what you want to evaluate at the earliest stages of project development.
