What was the programme and what did it aim to do?
This study evaluates two UK Government programs targeted at the long-term unemployed: Employment Training (ET) and Employment Action (EA). Between 1988 and 1993, ET and EA were the two main programmes in Britain for helping the long term unemployed (>6 months) back into work. The ET programme involved an average 6 months of training and/or job placements with employers. The EA programme involved work experience, often with voluntary organisations. These were large programmes together with around 150,000 participants at any one time and a combined budget in 1992-1993 of about £750m.
What’s the evaluation challenge?
The fact that participation was voluntary makes it harder to assess the programme’s impact. Individuals who decided to enrol will likely have different characteristics to people who did not enrol, which can bias estimated effects of the policy. For example, if participants were generally younger than non-participants then differences in duration of unemployment could simply be an effect of age. Unobservable factors, like attitude or ambition, could also determine who took part. Both observable and unobservable factors could explain reductions in unemployment duration which would then be wrongly attributed to training.
What did the evaluation do?
The authors used a matching approach to estimate the impact of the policy. A control group was constructed by carefully matching people to program participants on the basis of sex, age, local area and prior unemployment. Further characteristics were controlled for such as education, ethnicity, and whether the individual has problems with literacy and numeracy. The study then compared the change in employment outcome for those on the programme to those in the matched control group. Since this type of comparison involves two changes (or differences) it is known as a ‘difference-in-difference’.
How good was the evaluation?
According to our scoring guide, matching combined with difference-in-differences receives a maximum of 3 (out of 5) on the Maryland Scientific Methods Scale (Maryland SMS). This is because it does well to control for observable differences (e.g. age) between supported and non-supported individuals, but is unable to control for unobservable differences (e.g. motivation). Since this paper uses a wide range of variables in its matching we score this study 3 on the SMS.
To implement this approach, data for the treatment group was taken from interviews in the spring of 1993 with 1,674 people who had recently left ET or EA. They also interviewed a control group of people who did not use the program, selected from the Department for Education and Employment’s database of benefits claims. There were two follow up surveys at 12 and 24 months after the first survey.
What did the evaluation find?
The evaluation found that participation in ET increases the probability of exiting unemployment relative to the matched comparison group. This is true whilst the participants are on ET and after they finished the programme. However, the effect of EA is insignificant compared with those who did not take a programme.
What can we learn from this?
For policymakers, it is important to recognise the differences between the two programs to understand why ET apparently succeeded and why EA did not. The most striking difference is that ET offered training, whereas EA was purely a work experience program. However, this does not imply that training works and that work experience doesn’t work because (1) the ET program also offered work experience and (2) the EA work experience was very specific in that it was with voluntary organisations. These organisations are unlikely to be able to offer participants a job after the program finishes and the work experience they offer may not be as relevant for paid employment as that gained from a placement at a business. Therefore the success of the ET may be attributed to either the training component or the fact that the work experience component was with employers.
Firth, D., Payne, C., and Payne, J. (1999). Efficacy of programmes for the unemployed: discrete time modelling of duration data from a matched comparison study. Journal of the Royal Statistical Society Series a-Statistics in Society 162, 111–120. [Study 87 from our Employment Training review]