Insights
Our fifth economy recovery workshop focused on monitoring and evaluation (M&E). Covid-19 has changed how existing programmes are being evaluated, while also resulting in new policies, which were often developed at pace, limiting the time to think through how they would be evaluated. The workshop explored the challenges places are facing in undertaking good quality M&E. Rebecca Riley, from the West Midlands Combined Authority (WMCA), shared their experience, looking at the decisions they’d made about how to evaluate two policies – an employability programme that had to change its delivery model in response to Covid-19 and a business support programme that was developed in response to Covid-19. And our Director, Henry Overman, outlined his thoughts on the key issues that are affecting M&E including capacity, data availability, and disentangling the effects of Covid-19 from Brexit.
Another kind of monitoring – of the pace of recovery – is also now a necessity for local places. The workshop also touched on this, with Rebecca setting out how WMCA put in place a process early in the pandemic and how this is evolving in the recovery phase.
What are the main M&E challenges – and how might we overcome them?
The pandemic has made M&E more difficult in several ways. Local policymakers often have less capacity to undertake M&E than in the past. Given this, there’s a good case for focusing on meeting funding requirements and gathering data that will help make decisions about future programmes, such as whether to continue funding a project. Whilst impact evaluation is essential to understanding ‘what works’, given capacity restraints, in many cases the best option will be to capture good quality monitoring data. For example, the Pivot and Prosper programme was rapidly developed to help firms in the West Midlands develop new products and services in the early months of the pandemic. Good quality monitoring data allowed the WMCA to understand how this was affecting individual businesses and gave them the confidence to roll-out the programme regionally.
Another issue is that local policymakers need timely data to monitor and evaluate, and several workshop participants highlighted that this had become more of a challenge. The pandemic affected the availability and quality of several secondary datasets that local authorities routinely rely on, with some ONS surveys scaled back or scrapped in 2020. For primary data collected by local authorities themselves, changes in the ways the surveys are administered or difficulty in reaching some individuals has caused quality or comparability issues. Rebecca gave a good example of this with WMCA shifting from undertaking face-to-face interviews as part of an employability programme evaluation to telephone interviews. More positively, several new datasets became available during the pandemic. As well as being useful for examining the impact of policies, these can often also be used to control for differences in the impact of Covid-19.
Covid-19 has also created a challenge in terms of selecting an appropriate baseline year. There is unlikely to be a blanket answer to this – as it will depend on the question local policymakers want to answer. This challenge is compounded by the fact that Covid-19 effects are likely to be hard to distinguish from the impact of Brexit. There is no perfect way to address this, but it does reinforce the need to focus on selecting a good control group when evaluating policies.
How to monitor the economic recovery?
Having a good understanding of the pace and extent of the recovery is essential for local policymakers to help them target their efforts where they will have the greatest impact. Touching on this subject, Henry recommended that local policymakers focused on monitoring data on mobility, labour markets, and Covid-19. Each gives good insights into current economy activity, with timely data available. As all data sources have their limitations, he recommended that policymakers look at multiple indicators as this will help develop a clearer and more detailed picture about what is happening. You can find more details in our What Works Growth guide on monitoring the recovery.
Another key discussion point was the importance of linking monitoring to decision making. Rebecca outlined how WMCA quickly established a data collection and analysis process that fed into the weekly Economic Impact Group, chaired by the West Midlands Mayor. By collating a wide range of quantitative and qualitative data and insights, local policymakers were able to rapidly make decisions with confidence. Key to this was having a clear process about how the data would feed into decision-making. This process continues to form a key part of the region’s approach, with the evidence gathered helping inform the economic recovery.
The workshop highlighted how committed local policymakers are to conducting good quality M&E and proved a good forum for people to share ideas about how to overcome the challenges they face. As well as the guide to monitoring the recovery, we also have a range of how to evaluate resources that should help you when planning your M&E activities. We can also provide advice and support on impact evaluation – so if this is something that you are interested in, please do not hesitate to get in touch! And finally, if you’d like to watch the workshop in full, you can access a recording here.
