Evidence review
Statistical approach (SMS level 3)
What was the programme and what did it aim to do?
This study looks at the impact on firm performance of public support for innovation. Specifically, the study evaluates the impact of support by Innovate UK, the UK’s national innovation agency, funded by the UK government. Support is largely in the form of grants for R&D for innovative micro-enterprises or small and medium size enterprises (SMEs). A smaller part of the support involves loans or business advice. The grants are usually given on a competitive basis and are made available either to individual firms or as a stimulus for firm-to-firm or firm-to-university collaboration. The main objective of the support is to promote innovation. In turn, it is hoped this would help firms become more productive, increase output and hire more workers. Consistent with this, the study analyses the impact of support on measures of firm performance, specifically sales, number of employees, and survival.
What’s the evaluation challenge?
The evaluation of the impact of support for innovation is not straightforward as only certain types of firms are supported. For example, it could be that firms that are interested in the programme are more innovative and forward-looking. Furthermore, due to the competitive selection process, it may be that only the best performing firms are chosen to take part. Alternatively, the scheme might tend to support less well performing firms if these are more likely to apply for support. As a result of this kind of selection, better or worse outcomes for supported firms may simply reflect differences in the type of firm that is supported, rather than the impact of the programme.
What did the evaluation do?
The study created a control group of firms who were not supported, but who were similar to supported firms based on observable characteristics that affect the likelihood of being in the programme (e.g. previous employment, and sales). The technique used to do this is called ‘propensity score matching’. The study then compared the change in firm performance for supported firms to the change seen for the matched control group who did not receive support. Since this type of comparison involves two changes (or differences) it is known as a ‘ difference-in-difference’.
How good was the evaluation?
According to our scoring guide, matching combined with difference-in-differences receives a maximum of 3 (out of 5) on the Maryland Scientific Methods Scale (Maryland SMS). This is because it does well to control for observable differences (e.g. sales) between supported and non-supported firms, but is unable to control for unobservable differences (e.g. motivation in applying to the programme). Since this paper uses a wide range of variables in its matching, and since the difference-in-difference is based on a clear treatment dates, we score this study 3 on the SMS.
What did the evaluation find?
The study finds that supported firms increased their employment (by 32 employees, or about 11-14%), and were more likely to have survived (by 14 percentage points) four years after support began, compared with similar unsupported firms. The study also presents some evidence for increased sales (around 12-25%) resulting from support, although this result was not robust to using an alternative dataset. The employment and survival effects were largest for younger and mid-aged firms (2-5 years old and 6-19 years old, respectively), whereas the tentative sales effects were larger for mid-aged and older firms (more than 20 years old).
What can we learn from this?
When we undertook our evidence review in October 2015, we only found one UK study on grants and loans that met our evidence standards. So it is great to see a second study add to the UK specific evidence base. The finding of a positive effect on employment and weaker evidence on sales is in line with the other studies covered in our evidence review. The study suggests that the strong employment impact combined with a lack of robust sales impact could reflect different timing of effects. Firms may quickly hire more employees to carry out innovation tasks, but the eventual impact on sales of increased innovation may take longer to observe. Further studies using a longer timeframe may be needed to confirm if the employment effects persist (once staff cease to be funded by the grant or loan) and whether positive impacts on other measures of firm performance such as sales, or even productivity eventually emerge. Finally, it’s interesting to note that the employment effects that are detected are bigger for smaller and medium sized firms. Again, this is a finding that is in line with that of our evidence review.
Reference
Department for Business, Energy, and Industrial Strategy (2017), “The impact of public support for innovation on firm outcomes”, BEIS Research Paper Number 3, March 2017. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/604841/innovation-public-support-impact-report-2017.pdf
