Isolated headache is not a reliable indicator for brain cancer

Editor – Readers of the paper by Dr Ceronie and colleagues may be a bit baffled by discussion of the National Institute for Health and Care Excellence (NICE) criteria for 2-week-wait (2WW) referrals for suspected CNS malignancy.¹ For these criteria, which essentially constitute a screening test, NICE deemed a positive predictive value (PPV) of at least 5% to be preferable and adopted a PPV threshold of 3%.

Surely any clinician proposing a new screening or diagnostic test whose performance in a test accuracy study produced a PPV of somewhere between 0.03 and 0.05 (ie a false discovery rate between 0.97 and 0.95) would be laughed out of court, or sent for remedial training, or more likely would conclude that this was a negative study and consign the said test to oblivion without publication.

The issue, of course, relates to the dependence of predictive values on disease prevalence, which is very low for brain cancer in 2WW cohorts. Stated another way, the prevalence of ‘no cancer’ is very high. As a consequence of the large number of disease negative patients in any study of these criteria, since headaches not due to brain cancer vastly outnumber those due to brain cancer, there is over-inflation of test metrics, such as false discovery rate and negative predictive value.

The problem of accounting for excess correct ‘non-events’ has been recognised since the 19th century, specifically in the context of predicting tornados.² To allow for this, various metrics that eschew true negatives have been developed, such as the critical success index (CSI) or threat score, and the related F measure.

CSI = 1/[(1/PPV) + (1/sensitivity) – 1]

F = 2/[1/sensitivity + 1/PPV]

F = 2CSI/(1 + CSI)

These metrics range from 0–1, and higher values are better. They have already been applied in the assessment of cognitive screening instruments for the diagnosis of dementia in low-frequency settings (neurology-led dementia clinics).³ We recommend that they are adopted for the assessment of any screening or diagnostic test where prevalence of the condition being sought is low, accepting that this policy may require a broadening of clinician literacy in order to understand the meaning of these metrics.