How to evaluate business advice: Supporting unemployed people to start businesses (RCT)

What was the programme and what did it aim to do?

This study evaluates two US projects designed to support unemployed people to start their own businesses: the Washington State Self-Employment and Enterprise Development Project (SEED), and the Massachusetts UI Self-Employment Demonstration. Both programmes ran from 1989 to 1991, and tested whether public assistance could help participants move into employment or self-employment, raise individual incomes and increase business start-up rates. Both programmes provided a mix of expert training sessions, one-to-one sessions to develop business ideas and workshops covering business planning, marketing, accounting and management. In addition, the SEED programme gave participants completing the course a lump sum to help start their new firm.

What’s the evaluation challenge?

Evaluating the effect of business advice programmes is difficult because such programmes are used by specific types of firm. Typically, only certain types of firm will apply for support or be selected for support by the programme administrator. As a result of this selection, if we compare differences in outcomes for firms who received advice to those who did not, these differences may not reflect the impact of the programme. Instead, they may simply reflect differences in the types of firms who went on the programme.

What did the evaluation do?

Both programmes were designed as Randomised Control Trials (RCTs), with participants recruited from people receiving unemployment insurance. A random number generator was used to assign individuals to treatment or control groups, across a number of sites in each state. (In the case of SEED, 755 people were assigned to treatment, with a further 752 in the control group; numbers were slightly lower for the Massachusetts programme.) Randomisation makes it much easier to assess these programmes’ impact. Specifically, randomising receipt allowed the study to ensure the treatment and control groups are similar on observable characteristics (such as gender or qualifications) and unobservable characteristics (such as motivation) that might affect outcomes.

How good was the evaluation?

According to our scoring guide, an RCT receives a maximum of 5 (out of 5) on the Maryland Scientific Methods Scale (Maryland SMS). One potential problem with RCTs is people dropping out of the study – the researchers deal with this issue using administrative data, which allows tracking even if participants don’t respond to follow-up surveys. We therefore score this study ‘5’ on the Scientific Maryland Scale.

What did the evaluation find?

The study finds that both programmes increased the likelihood of entry into self-employment 21 months later, and accelerated moves from unemployment into self-employment (5.9 months earlier for SEED, 2.4 months for the Massachusetts programme). Both programmes also significantly increased the chances of being employed / self-employed, which was 14% higher for SEED participants and 5% higher for those in the Massachusetts initiative. However, neither programme significantly increased participants’ total earnings, in part because people were less likely to move into a regular job. The differences between the two programmes seem partly related to wider economic conditions in the two states; most of the US was in recession during the delivery period, but Washington’s economy was largely insulated from these macro trends.

What can we learn from this?

The studies provide very high quality evidence that business support interventions can be effective in raising employment, but less effective if the aim is to raise wages and incomes. What is less clear is exactly which elements of the treatment mix are most effective, and how it could be further fine-tuned. This is important to know, given the stronger performance of SEED may be related to programme design differences as well as external economic conditions. Further trials which randomised treatment strands would help answer these questions.

Reference

Benus, J.M., Wood, M.L., Grover, N., and Abt Associates (1994) A Comparative Analysis of the Washington and Massachusetts UI Self-Employment Demonstrations. Cambridge, Mass, Abt Associates. [Study 287 from our review on Business Advice]