Continuing the multiple sclerosis risk sharing scheme is unjustified
BMJ 2010; 340 doi: https://doi.org/10.1136/bmj.c1786 (Published 03 June 2010) Cite this as: BMJ 2010;340:c1786
- Christopher McCabe, professor1,
- Jim Chilcott, senior research fellow2,
- Karl Claxton, professor3,
- Paul Tappenden, senior research fellow2,
- Cindy Cooper, senior research fellow2,
- Jennifer Roberts, professor4,
- Nicola Cooper, senior research fellow5,
- Keith Abrams, professor5
- 1Academic Unit of Health Economics, Leeds Institute of Health Sciences, University of Leeds, Leeds LS2 9PL
- 2School of Health and Related Research, University of Sheffield, Sheffield
- 3Centre for Health Economics, University of York, York
- 4Department of Economics, University of Sheffield
- 5Department of Health Sciences, University of Leicester, Leicester
- Correspondence to: C McCabe c.mccabe{at}leeds.ac.uk
Since 2002 people in England with multiple sclerosis have been able to access disease modifying drugs through a risk sharing scheme.1 The scheme was set up after the National Institute for Health and Clinical Excellence (NICE) recommended that the drugs should not be used in the NHS because of doubts about their effectiveness and high price. It suggested instead that the Department of Health could work with the manufacturers to make the treatments available to NHS patients in a cost effective manner—that is, at a lower price.2
Under the terms of the scheme interferon beta (Avonex, Betaseron, and Rebif) and glatiramer acetate would be made available to NHS patients in the context of a study monitoring disease progression. The data were to be reviewed at two year intervals. If the observed benefit was less than that predicted by the model NICE had commissioned from the Sheffield School of Health and Related Research (ScHARR),3 which four of us worked on, the drug price would be reduced to achieve a target cost effectiveness ratio of £36 000 (€40 000; $54 000) per quality adjusted life year (QALY).1
Since the start of the scheme, 5583 patients meeting the Association of British Neurologist criteria have received one or more treatments, costing in the region of £350m. The first report of effectiveness was not published until last December.4 The authors of the report, which was published in the BMJ, concluded “we found no evidence that these treatments are cost effective.”4 They also state that “we cannot reliably determine whether the current pricing of these drugs represents value for money for the NHS.”4 Despite these findings, the NHS is still waiting for a price review. We examine the credibility of the reasons given for the delay and the implications for the NHS.
Monitoring data
The monitoring scheme compares the observed and expected disease progression using a deviation score. This score measures the difference between the expected and observed benefit from treatment expressed as a percentage of the expected benefit. It takes the value zero if the treatments perform as expected; greater than zero if the benefit is less, and less than zero if the treatment is better than predicted. Expected benefit is derived from a historical Canadian cohort followed up over 25 years, and disability was assessed with the extended disability status scale.4
The scheme established a tolerance range of 20% for the deviation score within which the treatments would be deemed to have performed as expected. The primary analysis reports a deviation score of 113%.4 This means that disease progression of treated patients was greater than expected for untreated patients.4 Patients who received the drugs are likely to have benefited from fewer relapses, but the drugs have not prevented any disability, and therefore the manufacturers would need to pay the NHS to use the drugs to make them cost effective.
Lack of action
Although the monitoring team concludes that there is no evidence these treatments are cost effective; it also argues it would be “premature, at this stage, to reach any decision about re-pricing the drugs without further follow-up and analyses.” It gives three reasons for this view: the validity of the assumption that patients’ disability cannot improve; that using historical controls is unsafe if the underlying epidemiology of the condition has changed over time; and that the cost and utility data used in the ScHARR model may have underestimated the impact of the disability avoided through treatment and thus underestimated the value of treatment.
All of these caveats were known at the start of the scheme. Given that they were not deemed sufficiently important to stop the scheme being launched, it is difficult to see how they can justify such an expensive divergence from the scheme rules. Perhaps more importantly, none of them supports the conclusion that the treatments are likely to be cost effective.
What if using historical controls is unsafe?
If the underlying rate of disease progression has increased since the control data were collected, the comparison used in the scheme would underestimate the effect of treatment. However, the literature on the severity of the disease suggests that although the disease is increasing in incidence, it may be less aggressive, rather than more so, in recent cohorts.5 This may be due to ascertainment bias, and the prudent presumption would be no change. An assumption of increasingly aggressive disease cannot be supported, and continuing with the scheme will not provide additional evidence of this.
What if the costs and utilities used in the model are wrong?
The ScHARR model applies the same costs and utilities to both treated and untreated patients in the same disease state—with the exception of the drug cost.3 The effect of inaccuracies in these data on estimated cost effectiveness depends on a divergence in the distribution of the treated and untreated cohorts across the scale used to measure disability. However, the observed treated patients do not diverge from the modelled untreated patients, so changing the costs and utility data used in the analysis cannot affect the estimates of cost effectiveness.
What if treatment really reversed progression of disability?
The BMJ paper reports that 32% of patients saw some improvement in their disability scores.4 However, the primary analysis discounted such improvements. If treatment really does reverse disability, it would be wrong to exclude this benefit from the assessment.
The authors suggest three reasons why recorded disability could improve between assessments—recovery from relapse, natural fluctuations, and measurement error.4 Neither recovery from relapse nor measurement error would represent real improvements in the underlying disability state. Therefore it is appropriate for disability reversals due to either of these causes to be excluded from the assessment.
Natural fluctuation is not defined by the authors, but we interpret it to mean real changes in domains of health measured by the extended disability status scale (EDSS), which although not severe enough to be diagnosed as relapse or recovery from relapse, could move some patients across EDSS thresholds. If fluctuations are more common among treated patients, it means treatment makes the symptomatic experience of the disease less rather than more stable, which would have a negative impact on the value of the treatment.
Research on the measurement of disability in multiple sclerosis is increasingly concluding that the EDSS is a poor measure of disability. Ebers and colleagues analysed the placebo arm data of more than 30 trials and concluded that none of the measures of disability used in the pivotal trials leading to regulatory approval, including the EDSS, were fit for purpose.6 Other authors have reported that early disease activity is a poor predictor of disease course and particularly time to major disability—that is, becoming wheelchair bound.7 Given these problems, the observed improvements in EDSS scores are more likely to reflect the limitations of the scale as a measure of disability in multiple sclerosis than any true benefit from therapy that should be considered by the risk sharing scheme.
Lessons for the National Health Service
None of the proffered reasons for delaying the price review withstand critical assessment. So why does the scientific advisory group consider a price review to be premature.4 The answer may lie in the governance arrangements of the scheme. The manufacturers, patient groups, clinicians, and the Department of Health are represented on the scientific advisory group. All these bodies have a vested interest in maintaining the status quo. The budget holders, who pay for these drugs, with responsibility for the health of populations served by the NHS, are not represented on the scientific advisory group, and as a result there is no countervailing influence on the group’s decision making. To be truly independent, the scientific advisory group should resemble the data monitoring committee of a clinical trial. Without such independence, hard decisions, such as recommending a price reduction or closure of the scheme, are unlikely to be made.
The delay in the publication of these results is a further cause for concern. Depending when the majority of patients started treatment, the first analysis has reported between two and five years after the data became available. The annual drug cost of the scheme is reported to be around £50m.8 If an assessment had been completed after the first two years, the NHS could have saved up to £250m already. Why has it taken this long? Again, the answer may lie in the governance arrangements. Day to day management of the scheme is the responsibility of the Multiple Sclerosis Trust, which campaigned for funding of these treatments and appealed against the NICE recommendation that these treatments were not cost effective.9 There is a tension between its historical position as a proponent of the value of these treatments and its current responsibility to deliver analyses that could lead to their withdrawal from NHS practice. Good governance requires that those charged with delivering these schemes are demonstrably free of such conflicts.9
The final lesson we draw from the scheme is for clinicians rather than government. At the time of the NICE appraisal, many clinicians considered it unethical to undertake a randomised controlled trial of these drugs.4 When decisions have no opportunity cost for other patients, such a position is defensible, but when treatment is funded at the expense of other patients’ health care, it is questionable whether it remains so. When the key uncertainty in the evidence base for a new product relates to its effectiveness, a randomised controlled trial is likely to be the quickest, most efficient, and most ethical strategy.
Notes
Cite this as: BMJ 2010;340:c1786
Footnotes
Contributors and sources: CMcC, JC, PT, and KA worked on the development of the ScHARR model used in the NICE appraisal and the risk sharing scheme. JC, PT, JR, CC, and KA were members of the academic team that set up the risk sharing monitoring study between 2002 and 2005. KC was a member of the NICE appraisal committee that considered the disease modifying therapies for multiple sclerosis. He also advised on the construction of the ScHARR model.
Competing interests: All authors have completed the unified competing interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that they have (1) no financial support for the submitted work from Merck-Serono; Schering, or Biogen; (2) CMcC worked on developing a previous cost effectiveness model for Serono over three years ago; (3) no spouses, partners, or children with relationships with commercial entities that might have an interest in the submitted work; and (4) See contributors and sources.
Provenance and peer review: Not commissioned; externally peer reviewed.