Insights
In my last blog I talked about the challenges of evaluating transport (and other capital investment). As I explained there, the key to high quality impact evaluation is to look at changes for those benefiting from a policy compared to a similar (control) group that don’t benefit. While it is fairly easy to understand how we might construct control groups and undertake evaluation for policies targeted at individuals, households or firms, it is harder to think about how we might do this for policies – such as transport investment – that target areas.
But at least we do quite a lot of transport investment, making the number of areas that benefit quite large. Is there anything that we can do to evaluate the impact of interventions – such as devolution deals – that target a small number of areas?
Unfortunately, our options are quite limited, but not zero. In a 2003 paper in the American Economic Review (one of the highest ranked peer reviewed economics journals) Alberto Abadie and Javier Gardeazabal propose a synthetic control methodology for looking at the impact of terrorism on the Basque country. With only one ‘targeted’ area it’s impossible to come up with an obvious control group. A similar problem would apply for devolution deals. Comparing changes in Manchester to those in London doesn’t seem right. But neither does it seem right to compare changes in Manchester to those in, for example, Birmingham, Sunderland or Hull. Abadie and Gardeazabal’s solution was to construct a synthetic control by weighting outcomes for a number of different reasons. The details on how to set the weights are pretty technical, but the principle is simple – we choose the weights so that the synthetic control is as similar as possible to the target area in the run up to the policy change. This can be either in terms of characteristics of the area – e.g. the industrial and skill composition – or in terms of the outcome of interest – e.g. GDP. After the policy is implemented we look at the change for Manchester relative to the change that occurs in the (appropriately weighted) control areas.
Could such an approach work for devolutions deals (and other area targeted policies)? I see two challenges. First, there are a number of areas signing devolution deals. This limits the areas that can be used to help construct the controls. Second, these approaches are still in development and a number of issues are not yet resolved. My own work on the impact of the Diamond Synchrotron discusses some of the challenges involved. All of this means that, in practical terms, such an approach is unlikely to be sufficiently credible to bear the weight of decisions on extending deals, rewarding areas for increased growth, and so on. I think it could be a valuable input into those decisions, but other work will be needed, especially given the political sensitivities involved.
I think it’s also important to highlight the conclusions that we reached in our study of European Union policy. The EU studies we were able to review clearly demonstrate the limits of evaluating multi-strand ‘area-based initiatives’ at a large regional scale. It is also essentially impossible to say anything on the cost-effectiveness of different types of expenditure from these overall evaluations. While a synthetic control approach might provide a useful input into decision making on devolution deals, we continue to recommend that the main focus should be on designing specific high quality evaluations of particular strands of expenditure, rather than attempting only a single overarching evaluation.
