The dark side of surrogate endpoints
Why not all mechanisms to cut costs in drug approval are worth it
In economics there is a relationship called Goodhart’s Law, after Charles Goodhart, LSE professor and onetime member of the Bank of England’s Monetary Policy Committee. Goodhart’s Law states that the relationship between any indicator of some phenomenon that is not the phenomenon itself and the phenomenon will break down when you try and exploit it.
The most famous example is the Phillips Curve. The Phillips Curve established a remarkably strong negative relationship between unemployment and inflation between 1861 and 1961. Based on this relationship, Keynesian postwar economic planners tended to try and fine tune aggregate demand to keep the economy at full employment. But as soon as they started doing this, the relationship started to break down. In the 1970s higher inflation tended to coincide with higher unemployment.
Recall the concept of ‘surrogate endpoints’ from my last post on drugs. Usually, the FDA approves your drug based on whether it affects the lifespan of patients. However, in some cases, it approves it based on surrogate endpoints – things that correlate with lifespan that we can track much earlier. This makes drug development, especially for long term conditions, dramatically cheaper to do.
Well-defined surrogate endpoints do achieve their goals. Low density lipoprotein (LDL) cholesterol seemed in the 1960s and 1970s to be a strong indicator of cardiovascular problems - heart disease, heart attacks and CVD deaths. Statins lower LDL cholesterol, and were approved by the FDA to treat those at risk of heart disease for that reason. Now, Goodhart’s Law did still operate: statins do not lower CVD as much as we would expect given how much they lower LDL. But it didn’t fully wipe out the link—there remained a correlation between LDL and mortality. With hindsight, now that we have large meta-analyses covering tens of thousands of those who have used statins we find they did indeed lower overall mortality.
Much more aggressive versions of Goodhart’s Law have operated when it comes to other surrogate endpoints. Oncology drugs can be approved both by the FDA and the EMA (its European equivalent) on the basis of tumour response rate (the percentage of patients whose cancer tumour shrinks after treatment) or progression-free survival (how many patients survive without their cancer growing).
Intuitively, and based on established correlations, these endpoints both correlate with overall all-cause mortality (and cancer mortality). However, in practice drugs approved on the basis of improving these metrics have not improved survival much—or in some cases at all. One obvious reason is that drugs can have extremely bad side effects, potentially in some cases outweighing or balancing out the benefits seen via the endpoints. In 2011, the FDA removed the breast cancer indication1 it had given to Avastin (bevacizumab), citing safety concerns—at this point it had also shown little impact on overall survival.
Chul Kim and Vinay Prasad review all 36 cancer drugs approved in the USA between 2008 and 2012 on the basis of surrogate endpoints. Of these 36, only five were shown to improve survival at all. Eighteen did not improve survival in the follow-up trials; and at the time of their paper (2015) a further 13 had unclear effects on overall mortality. The same authors, with Mauricio Burotto, followed up this work by systematically reviewing and meta-analysing the literature on these oncology surrogate endpoints. They found in 2015 that most studies of current oncology surrogate endpoints have a low correlation with overall survival. A 2019 follow-up review found similar results.
This has combined with perverse funding mechanisms to create some severely distorted incentives for research. Medicare, which funds healthcare for around 60 million Americans, and nearly all Americans over 65, is obliged to fund any cancer treatment that gets FDA approval. Combined with the fact that cancer drugs can be approved on the basis of weakly validated surrogate endpoints, and you have a perfect storm where relatively unhelpful drugs are paid for - whatever the price. Critics of the system like Vinay Prasad allege that only fear of public backlash restrains firms from setting essentially infinite yearly prices on their drugs. Worse still, firms know this, and are incentivised to spend heavily on researching new marginally effective oncology drugs.
A final problem is that this effective loosening of standards, while not necessarily a problem if combined with rational funding mechanisms, is distorting our overall data on the number of new molecular entities approved in given years.
For decades, the numbers of drugs approved per billion dollars spent has been falling—this is called ‘Eroom’s Law’. Eroom’s Law is the inverse of Moore’s Law, which describes the consistent and steady progress we have made on fitting more transistors on a computer chip. It describes the consistent and steady decline in the number of new molecular entities approved by the FDA per billion dollars spent by Big Pharma.
Since the late 00s there has been a huge increase in the number of drugs designated orphans (for rare diseases), as well as oncology drugs approved based on surrogates. This recent uptick has driven a flattening of the otherwise endless Eroom’s Law seen for previous decades. An Eroom’s Law with QALYs saved per $1bn drug R&D on the y-axis would not show this flattening of the trend. In a sense, Scannell’s metric itself suffered a Goodhart’s Law style breakdown as soon as it began to be measured.
So what can be done? I don’t personally think the issue is so simple as surrogates bad, survival good. I think the issue is that surrogates are approved and validated in a much more arbitrary way than drugs themselves. Drugs go through the most rigorous scientific process of pretty much anything in our whole society. They apply basically every rule and procedure suggested by advocates of good science like Stuart Ritchie: pre-registration, randomisation, multiple centres, open results, and so on. But the FDA approves surrogates in a more qualitative way.
Why not require surrogates to go through a similar procedure to drugs – proof of their meaningful link to survival through rigorous tests – but then allow drug companies to patent and license the surrogates for a time. Surrogates can be tested in randomised controlled trials with another main outcome, but as a pre-registered secondary outcome. And surrogates can be tested through other non-RCT but quasi-random methods, for example Mendelian Randomisation.
In this way, they would have an incentive to discover and prove the most valuable links, and we could dramatically cut costs and speed up drug development, without generating a whole range of drugs that don’t work, like the current system.
P.S. Works in Progress has just released its first ever ‘special issue’, which tackles the question ‘Dude, where is my flying car?’
When the FDA gives a drug an ‘indication’, it means that doctors can prescribe it for people affected by that condition.