Evidence review|Evidence briefing
What was the programme and what did it aim to do?
This study evaluates the impact of two social landlord interventions on the health and wellbeing of people aged over 50 living in socially provided “general-needs accommodation” in London. Two services were tested: a signposting service where participants were referred to a suite of health and wellbeing interventions and a more intensive ‘handholding’ service provided by a specialized in-house team of health and wellbeing support workers.
This study focuses on health outcomes but holds valuable lessons for local economic growth interventions.
The delivery of the trial relies on the local knowledge and network of social landlords to increase take-up rates of the more disadvantaged groups. The issues faced by this trial thus carry over to those employment-training RCTs interested in this delivery mechanism.
In addition, good health plays a major role in the ability of the more disadvantaged to get and retain employment. The findings of this paper thus speak more broadly to the types of challenges that local economic growth RCTs might face when targeting support to poorer individuals or households.
What’s the evaluation challenge?
Evaluating the impact of social landlord interventions such as these is difficult because people who choose to participate in these interventions and receive support tend to be different from those who do not in ways that are hard to observe or measure. As a result of this selection, if we compare differences in outcomes between those enrolled and those not enrolled, these differences would not necessarily reflect the impact of the program. Instead, they may simply reflect characteristics of participants who selected themselves into the program (e.g. a more disadvantaged demographic, greater health needs, a better relationship with their landlord). The evaluation challenge is thus to isolate the impact of the program from the unobserved characteristics of those taking up the program.
What did the evaluation do?
The study deals with these unobservables by implementing the pilot as a randomised control trial (RCT). 547 participants took part in this trial for a period of 18 months on average. Eligible participants had to be 50 years or more, living in General Needs Family Mosaic property (the social landlord partner) and within the borough of Hackney, Islington, Hammersmith and Fulham, Kensington and Chelsea or Haringey. The evaluation exploits the randomized provision of support whereby participants were drawn randomly from a social housing association with a substantial waiting list. A third of the participants were offered the light-touch signposting service, a third were offered the more intensive ‘handholding’ service and the remaining third were used as the control group. If done correctly, randomisation should ensure all the individual characteristics are balanced out statistically between the control and treatment groups such that on average, the mean estimated difference in performance between the treated and the non-treated cannot be attributed to personal characteristics but instead captures the impact of the programme.
The evaluation compares the difference in health outcomes for the treated before and after the service was offered, to the difference in health outcomes for those who did not receive the service during the same period.
How good was the evaluation?
According to our scoring guide, randomised control trials can achieve the maximum score of 5 on the Maryland Scientific Methods Scale. This is because randomisation balances out the observable (baseline health) and unobservable characteristics between treated and non-treated applicants. In this study, the randomization worked satisfactorily: the two treated groups and the untreated group were not statistically different on all observables collected at baseline.
Another important challenge in an RCT is attrition, especially differential attrition. Differential attrition occurs when attrition rates vary between treatment and control groups. Given the characteristics of those who stayed on the program might differ from those who do not in ways that interact with the outcome of interest, this could introduce a bias between the two groups that is not attributable to the treatment per se. The study finds that 77% of the original sample was retained through to the point of final assessment. Although attrition was higher amongst the control group, the authors found that there were no statistically significant differences in attrition rates between groups at the 5% level in terms of demography, gender or ethnicity.
A limitation of the study was the inability to access NHS patient data, making it necessary to rely on self-reporting. There may also be concerns about sample selection, with those with greater subjective health needs being more likely to participate. On other measures, the sample was representative of the broader over-50s general-needs London tenant population making the generalizability of the findings strong.
Overall, we score the study at the maximum of 5 on the SMS.
What did the evaluation find?
Quite minor health interventions involving guidance from support workers generated non-statistically significant improvements on a range of outcomes. The only participants for whom outcomes were found to be statistically significant were the most intensively treated group. These participants had no GP or contact with external medical help despite carrying urgent at times life-threatening diseases. For this particular group of participants within the treatment group, significant reductions were found for 1)planned hospital usage; 2) nights in hospital; and 3) for emergency GP usage. In contrast, there was no evidence of a positive effect from the signposting intervention: there was even some indication that this intervention might have had negative impacts on wellbeing, by raising expectations of participants in the trial and not meeting those expectations with the service offered.
What can we learn from this?
We draw two mains lessons from this trial, which carry over to local economic growth trials.
Drawing a sample from disadvantaged groups in social housing in London revealed 4.5 percent of the population had very serious untreated health problems. Economic growth trials should anticipate these challenges when appraising the type of economic support provided to this demographic. Some participants have felt frustrated as a result of becoming aware but not feeling able or confident to actually take advantage of services that are potentially helpful to them. Such potential harmful effects must be carefully considered for future health but also economic growth trials and weighted against the potential benefits.
The trial also tells us that intensive support was found to increase take-up and significantly reduce NHS usage for a target group (older and poorer people who had no prior contact with a GP) compared to a light touch support. However, this paper confirms findings of other studies in the literature, which indicate that identifying significant outcomes when evaluating community-based interventions for older people is challenging. Future health and economic trials should keep in mind the difficulty of estimating statistically significant outcomes with such trials. Randomisation helps, but does not totally solve, the problems of achieving sufficient statistical power given the small order of magnitude of program impacts in this field and for this target demographic. Equally the use of social landlords helps but does not totally solve problems around retention rates (77%).
Reference
Cheshire PC, Gibbons S, Mouland J Social tenants’ health: evaluating the effectiveness of landlord interventions J Epidemiol Community Health Published Online First: 08 March 2018. doi: 10.1136/jech-2017-209888
