Insights
There are a number of big barriers to improving the use of evaluation in policy design. Our how to evaluate series focused on a few of these and the reaction to the series suggests that people have found these helpful. So we’ve been thinking recently about how to further develop that work. One thing we’re keen to do is to provide some practical guidance on implementing Randomised Control Trials. But reading around the issue has also highlighted a number of things we might add to the original how to evaluate series.
For example, reading a recent paper by Bill Kerr and colleagues on ‘Entrepreneurship as Experimentation’, I was struck by their discussion of when to evaluate. Whereas we’d thought about that as a question of timing, they’re interested in the question about whether or not to undertake an evaluation at all. They argue that experimentation is particularly valuable in cases where initial information can be especially informative about the overall quality of the project and this information is cost-effective to obtain. This has obvious implications for the question of whether to experiment for a whole host of local economic growth policies (something that I’ve talked about more in our work on demonstrator projects.
Another issue discussed in the paper is the importance of being able to terminate projects if the experiments don’t work out. Both these points argue against large sprawling evaluations of big projects that occur well after plenty of money has been spent. But even when those big evaluations aren’t particularly positive, it can be hard to stop expenditure on unsuccessful programmes. Early evaluation certainly helps mitigate some of the problems (e.g. the need to defend large amounts of expenditure), but curtailing a pilot on the basis of a negative evaluation can still be very difficult – as this story from the Alliance for Useful Evidence makes clear.
For entrepreneurs, these problems are compounded by financing risk – i.e. needing to go back to the market for more funding to help scale up ideas. While it would be nice to think that government policy shouldn’t exhibit this problem, in practice the need to keep going back for additional financing may be an additional barrier (particularly in the context of devolution and debates about the split of powers between national and local).
Finally, Kerr and co-authors note the importance of being able to document and transmit the results of experiments so that findings can be acted upon. This will be particularly important in the local economic growth context, where policy lessons need to be transmitted across local areas.
We’ll be looking to expand on all these points in our future work.
