76x cash transfers: too good to be true or a (missed) opportunity?

This post was orginally posted on the EA Forum. It was adapted for posting here.

‍

Introduction

GOAL 3 commissioned the Rethink Priorities Global Health and Development team to provide an external analysis of the impact and cost-effectiveness of our product ‘IMPALA.’ IMPALA is a continuous patient monitor combined with a digital health platform designed for low-resource settings. The report estimates that IMPALA is highly cost-effective, far surpassing every funding threshold considered. 

‍

Understanding the problem in its context

Sub-Saharan Africa accounted for 2.8 million of the 4.9 million deaths of children under five worldwide in 2024. Access to care alone does not prevent these deaths. In low- and middle-income countries, an estimated 58% of all amenable deaths from conditions treatable by health care occur among people who did use the health system but received sub standard-quality care. Root causes to this challenge are lack of adequate equipment and shortage of staff.

‍

In many low-and-middle-income countries (LMICs) there is a huge shortage of staff. WHO uses 4.45 doctors, nurses and midwives per 1,000 people as an indicative minimum for progress on the health-related Sustainable Development Goals. A survey of 47 countries found that the WHO African Region had an average of 1.5 doctors, nurses and midwives per 1,000 people, about a third of WHO's minimum (Malawi was even below 0.5, less than a ninth). Moreover, staff lack the equipment they need to monitor and manage their patients with up to 40% of equipment malfunctioning. Qualitative studies in paediatric high-dependency units in Malawi describe monitoring as intermittent, with nurses forced to focus on the sickest children while staff shortages, power cuts and too few working devices get in the way.

‍

In these conditions, health workers are often unable to recognize and act upon patient deterioration, forcing them into reactive care; leading to late stage response to emergencies when treatments are more costly and more time consuming while outcomes are poor. This sets off a vicious cycle whereby scarce time and resources are used on late-stage escape treatments, further hampering the ability of the health systems to respond in time to changing patient conditions.

‍

Theory of Change

IMPALA enables health workers to monitor patients continually with health workers receiving alerts when a patient deteriorates. As a result, health workers can recognise patient deterioration earlier and prioritise care more effectively. Monitoring also reduces staff workload. By automating the process of checking vital signs, IMPALA helps save hours of work for health workers every shift, allowing them to focus their attention where it is most needed. 

‍

Additionally, when patients are treated earlier, complications can be prevented and treatment demands fewer resources. This means less medicine use, less patient costs, and reduced time spent on treating the child by the health workers. The result is reduced health expenditure. See Figure 1 for our working diagram of the Theory of Change for IMPALA. 

‍

Importantly, IMPALA is not an additional intervention that increases the burden on already overworked health workers but is embedded into their workflow. IMPALA is built into health workers' existing routine rather than added on top of it. Each implementation includes on-site training for the ward team, a group of trained IMPALA champions among the hospital's own staff who support colleagues and onboard new starters, and ongoing follow-up from our in-country team through IMPALA Care. 

‍

What the RP report tells us

Rethink Priorities' Global Health and Development team took on the challenge of evaluating our evidence. RP reviewed nine mortality estimates from hospitals in Malawi, Rwanda and Tanzania. Eight of these point towards lower mortality.

RP's reading is that our evidence consistently points to a mortality benefit but does not establish its size, so its model discounts heavily. RP combined our evidence into weighted averages which led to the number of 42% mortality reduction for paediatric wards and 26% mortality reduction for neonatal units. It then applied a 60% internal validity discount and a 30% external validity discount, in line with how GiveWell treats non-randomised evidence. That left a modelled effect of 12% mortality reduction in paediatric wards and 7% in neonatal units.

‍

RP then estimated cost-effectiveness using its full model (see full model parameters and results here). It puts the full cost to the health provider at $6,100 per ward per year (annualized over IMPALA's lifetime), and assumes 3,000 admissions per year on a paediatric ward and 1,000 on a neonatal unit. On that basis, it estimates that IMPALA costs $7–17 per DALY averted in paediatric care and $12–30 in neonatal care. This translates to around 76× and 28× unconditional cash transfers against GiveWell's bar, and roughly 11,700× and 4,400× on Coefficient Giving's scale against their 1,000× bar for pediatric and neonatal care respectively. The full table from the report can be seen below in figure 2. 

‍

‍

Interpretation, strengths and limitations

A model that returns 76x cash transfers could make any reader sceptical and we understand that.  It is important however, to root this in understanding of the intervention and its context. The intervention works (in contrast to many public health interventions like malaria nets or vitamin A) within an environment in which already many resources are deployed. IMPALA allows hospitals to optimize the use of these scarce resources for the patients' benefit by automating repetitive tasks and ensuring essential information is readily available. Because patients have less complications and go home faster it further relieves pressure on the health system which benefits quality of care but also contributes to workload reduction and cost savings. This was found in a cost-effectiveness study  recently published in the BMJ Paediatrics Open. We like to think of IMPALA as adding oil to a rusty machine. All components are already there, we just allow them to work more effectively together.

However, we want to be clear about what the evidence does not yet show. 

‍

There is no randomised trial as part of our evidence. Most estimates compare a ward before and after IMPALA, so anything else that changed between those periods can look like an IMPALA effect. In our only controlled study, economic and health effects (incl. cyclones, drought, and economic crises) outside of our control may have influenced the results. As the study had a single control per site, RP notes, a shock at that one facility is hard to separate from a real trend. 

‍

Several estimates also show little or no effect of IMPALA. Their confidence intervals cross zero, or, the lower bound sits at a 1-2% reduction. Furthermore, large effects measured in small samples tend to shrink as more data comes in. GOAL 3's service model includes ongoing support, which may preserve some of that, but we cannot yet say how much. GOAL 3 has also published its non-significant results and has been transparent with all the data, including one hospital where mortality did not change, which RP took as a sign that they were assessing the full body of evidence rather than a favourable selection.

‍

Simultaneously, we still think the case holds up. As RP notes: “A consistent direction across independent implementations is harder to explain by chance than any one result considered alone.” The body of evidence is wide enough to show that IMPALA consistently points in the direction of mortality reduction across settings.

‍

The starting point also matters. Standard care in the Malawian HDU in these studies is manual observation every six hours. A study in Kenya found hospitals completing only about half of the intended manual checks. Against that baseline, a modest improvement is far more plausible than it would be in a well-resourced hospital.

‍

Finally, the cost-effectiveness calculation from RP does not depend on a large effect. Implausible cost-effectiveness figures usually come from implausible effect sizes. Here the modelled effects of 12% and 7% sit well below every favourable site estimate. As RP also noted, “We believe this is a robust conclusion because it is driven by low cost rather than a large effect: the break-even mortality reduction is only ~1% (pediatric) and ~2% (neonatal), so the conclusion holds even if there is only a very minor mortality benefit.”

‍

For IMPALA to fall below GiveWell's bar, it would have to save almost no lives, and the evidence points far in the other direction.