Insights
Well-designed evaluations that clearly demonstrate the economic impacts of local policy interventions are not easy to come by – as anyone who follows our work will know. Although we’re helping grow the evidence base, it’s still relatively rare to find an evaluation that meets the Centre’s highest evidence standards and demonstrates significant positive impacts. The latest evaluation from an employment training programme in the US has caught our attention as it ticks both these boxes, and we’re keen to see if it can be replicated in the UK.
‘Gold Standard’ evaluation
The Per Scholas WorkAdvance programme, which delivered IT training and employment support for low-income individuals in the Bronx, has been evaluated through a rigorous randomised control trial. The evaluation is a relatively large one: 2,564 individuals were enrolled in the scheme and then randomly assigned to either the WorkAdvance programme (690 of 1,293 enrolled on to the Per Scholas programme) or the control group (1,271). Random assignment ensured there were no systematic differences between the two groups – and that outcomes could be attributed to the programme. Drop-out rates are low, and outcomes have been tracked through surveys (80 per cent of the sample were interviewed in the Year 2 survey) and administrative data on wages for all sample members. This gives us relatively strong confidence in the findings.
Significant earnings gains
Three years on from the programme, Per Scholas has been found to have ‘large and growing’ impacts on employment and earnings, with participant earnings $4,800 or 27 per cent higher on average than the control group. This is an increase on the two previous years, with an earnings effect of zero in year one to $3,700 in year two. Both effects were statistically significant. While the full cost-benefit analysis is yet to be published, it appears that it has also been cost effective. The operating costs per participant were $5,700 at Per Scholas. The net costs – operating costs minus costs of similar services received by the average control group member – were $3,500 per Per Scholas participant. If earnings gains, coupled with savings to the public purse, are sustained, then the benefits are likely to far outweigh the costs of the programme. And it mirrors another RCT conducted on a Per Scholas programme in the Bronx eight years earlier, which led to a 32 per cent increase in earnings. The chances of both being a statistical fluke is very small and so adds further weight to these latest findings.
Core features of the programme
The programme has five core components which are sector-specific in design, which includes: Intensive screening of applicants for motivation, capability and need; pre-employment and career readiness services; occupational skills training that meets the needs of local employers (15 weeks at Per Scholas); job development and placement services based on strong relationships with employers; and post-employment retention and advancement services. Each component required providers to be more employer-facing than traditional programmes – one of the key design elements identified as helping improve effectiveness in our review of employment training.
Lessons learned
The WorkAdvance programme was delivered by three other providers alongside Per Scholas and the effects varied widely across them: while Per Scholas delivered earnings gains of $4,800, one provider delivered none. So what was it that set Per Scholas apart?
Each of the four providers targeted different sectors and some have questioned whether Per Scholas’ focus on the IT sector as opposed to sectors targeted by other providers, which included environment remediation, transport, manufacturing and health care, has had a bearing on their ability to deliver such remarkable earnings gains. Yet there was no notable variation in sector premium between sites. More important, the evaluation argues, is that Per Scholas was by far the most successful at increasing employment in its targeted sector: participants were three times more likely to be working in the IT sector than those in the control group (twice as large as the impact by any other provider).
Their success is thought to be related to a number of factors. Firstly, Per Scholas has substantial experience in providing IT training and has been delivering WorkAdvance programme components since 1998. Related to this was Per Scholas’ ability to adapt its curriculum and training offer based on employer feedback. They also delivered all training in-house which gave staff more opportunity to interact with participants and deliver career readiness services.
More widely the WorkAdvance programme highlights the importance of monitoring and evaluating from the beginning and over the longer term. Evaluation was built in from the start and because of this it became clear fairly early on that those on ‘placement first’ programmes were entering into low paid jobs, not gaining the skills needed to progress in work, and providers changed tack as a result. The study also demonstrated that it can take at least two years for the economic impacts to emerge, reinforcing the need for longitudinal analysis.
Replicating this level of success in the UK
With stagnating wages and the rise in in-work poverty, it’s critical that UK policy makers find ways to support individuals into good jobs with opportunities for progression. There’s lots of policy work on-going in this space (with arguably more focus on apprenticeships in the UK) but it remains to be seen if any UK providers have managed to deliver results as significant to Per Scholas.
