ABSTRACT
Patients and public have sought mortality risk information throughout the pandemic, but their needs may not be served by current risk prediction tools. Our mixed methods study involved: (1) systematic review of published risk tools for prognosis, (2) provision and patient testing of new mortality risk estimates for people with high-risk conditions and (3) iterative patient and public involvement and engagement with qualitative analysis. Only one of 53 (2%) previously published risk tools involved patients or the public, while 11/53 (21%) had publicly accessible portals, but all for use by clinicians and researchers.
Among people with a wide range of underlying conditions, there has been sustained interest and engagement in accessible and tailored, pre- and postpandemic mortality information. Informed by patient feedback, we provide such information in ‘five clicks’ (https://covid19-phenomics.org/OurRiskCoV.html), as context for decision making and discussions with health professionals and family members. Further development requires curation and regular updating of NHS data and wider patient and public engagement.
Introduction
Coronavirus disease (COVID-19) has caused unprecedented personal interest in mortality risk,1–4 particularly among individuals with underlying health conditions which policy makers consider high risk.2,3,5–7 These high-risk conditions span clinical specialities, each with differing approaches to generating and providing information on prognosis to patients.8,9 Mortality data for such a wide range of conditions have never been generated and made available to the public.2 COVID-19 provides an opportunity, perhaps a responsibility,10 to bring patients and public into the generation of new knowledge on risks, determining how risk information may be brought together and provided for the use of patients.11,12 PROGRESS guidelines on prognosis research have recommended greater patient and public involvement and engagement (PPIE) in the ‘goals and value of prognosis research’.13
PPIE faces challenges due to COVID-19.14–16 Risk tools have emerged,17–21 but may not directly respond to patient and public concerns, or provide publicly available baseline mortality data across a wide range of conditions.20 Therefore, public needs, utility and acceptability of providing mortality risk data, tailored to underlying conditions, age and sex, are unknown. Prepandemic, PPIE has been criticised as ‘too little too late’ and difficult to demonstrate impact.22
Our objectives were to: (1) systematically review PPIE in development of mortality risk tools and availability and use of patient-facing mortality risk information, and (2) iteratively develop, with patients, accessible mortality risk information for ‘high-risk’ conditions for COVID-19.
Methods
Our mixed methods research was formative and iterative, responding to changing patient and public concerns as pandemic and policy evolved, beginning in March 2020.3,5,6
Systematic review of COVID-19 risk prediction tools
We began our search for public-facing COVID-19 risk prediction tools for severity or mortality to inform our preprint on risk in underlying conditions before first UK lockdown, posted on 22 March 2020.23,24 Based on a living systematic review of COVID-19 risk tools,20 we updated PubMed and Google Scholar searches for COVID-19 prognostic models on 4 February 2021. We included studies regardless of language, setting or publication status. We extracted data regarding publication (peer review or preprint), model (eg number of patients, variables, underlying conditions and outcomes in derivation dataset), PPIE (methods and reporting), publicly available portals (including information about user numbers), baseline and observed mortality risk.
Generation and presentation of observed mortality risks in underlying conditions
We generated and presented tailored mortality risk data for underlying conditions. We gauged broad views on design, content and use of mortality information, leading to prototype release on 12 May 2020 with six conditions and later development with 87 conditions, presented here.
Our methods to generate observed and modelled mortality risks are published.25 We used population-based primary care electronic health records (EHRs; Clinical Practice Research Datalink: CPRD-GOLD). Eligible individuals were aged ≥30 years and were registered with a general practice between 1 January 1997 and 1 January 2017 with ≥1 year of follow-up data. We defined each ‘moderate-risk’ (also ‘clinically vulnerable’) and high-risk (also ‘extremely vulnerable’) conditions listed by NHS England3 using open, reusable definitions of diseases from the Health Data Research UK (HDR UK) CALIBER phenotype library (www.caliberresearch.org/).26 We defined prevalent underlying conditions as present at any time prior to baseline, including EHRs from 1 year prior to baseline until date of death or 1 January 2017. We analysed data for 3.8 million patients with 48,964 deaths within 1 year.
For each condition we generated observed (unadjusted) risk using Kaplan–Meier estimates of 1-year all-cause mortality in each age, sex and ‘more than one listed condition’ (multimorbidity) cell to calculate excess deaths by age and sex bands, and number of conditions (1 or ≥2). In each underlying condition (by age and sex), for people with multimorbidity, we took average mortality for ≥2 conditions. To allow the user to explore potential impact (direct and indirect effects) of the pandemic on risk, we include an adjustable relative risk (RR), using previously published condition-specific RRs or hazard ratios for each disease for fatal COVID-19,27 under the assumption that they may also be relevant to all-cause mortality. Analyses were carried out using R (version 3.4.3), and the calculator implemented in R ShinyTM. Study approval was by MHRA (UK) Independent Scientific Advisory Committee under Section 251 (NHS Social Care Act 2006; protocol [20_074R]), working with stakeholders to develop a public-facing online tool to communicate risks.
Patient and public involvement and engagement
PPIE informed two phases: until prototype release (12 May 2020)28 with publication,25 and later development into public-facing version. There were five types of PPIE: (1) four rounds of individual written feedback from a patient and public panel convened by HDR UK (‘HDR UK PPIE panel’) (n=39 of 51 members)29; (2) informal and structured, written feedback from stakeholders; (3) user and media feedback on the prototype; (4) three virtual focus groups (total n=15 participants; five independent of HDR PPIE panel); and (5) specialist engagement. Using written transcripts, feedback and comments, we conducted inductive–deductive thematic analysis.30
Informing prototype (March–May 2020)
With five members of the public, by 28 March, we designed the risk tool concept version. The first round of written feedback via the PPIE coordinator (SM) was on 27–30 March with the HDR UK PPIE panel (39/51 members; 69.6% response rate; 17 males), 30 of whom have ≥1 at-risk conditions for COVID-19. We obtained informal and structured feedback from stakeholders: patients (independent of HDR UK panel; n=7), UCL (clinicians, data scientists and informaticians; n=30), and non-UCL(researchers and clinicians; n=15).
Informing subsequent development of public-facing version (May 2020–November 2020)
After prototype release in May 2020, we collated user analytics of the R shiny tool, user emails (n=147), media and journal coverage. We had three further rounds of HDR UK PPIE panel feedback (n=36, n=34 and n=34 of the original 39). In addition, we invited PPIE panel members to join three virtual focus groups to refine the tool and discuss feedback: UCL Institute of Health Informatics (11 July 2020; n=5, independent of other patients and public at other stages on); and HDR UK (13 and 14 August 2020; n=9 and n=10). Following PPIE in September, we held face-to-face virtual meetings with policymakers, charities, patient organisations and clinicians (11 separate meetings; n=20) to provide further insights on potential role of public mortality risk information.
Ethics approval
The study was approved by the MHRA (UK) Independent Scientific Advisory Committee under Section 251 (NHS Social Care Act 2006), with a protocol [20_074R], which set out working with stakeholders to develop a public-facing online tool to communicate risks.
Results
Systematic review of COVID-19 risk prediction tools
We identified 53 published COVID-19 risk prediction tools (including OurRisk.CoV; five additional to the living systematic review), including 29.1 million patients (61,280 outcomes) from 76 countries (five from multiple countries) (see supplementary material S1, Fig 1). Excluding OurRisk.CoV, 1/53 (2%) published risk tools included a statement about PPIE but did not detail methods.18 Eleven studies (11/53, 21%) provided publicly available portals but none were primarily for use by patients or public, or provided information on use, user numbers or user feedback. Median (range) number of included high-risk conditions was 16 (5–64) in general population studies (n=5), and 1 (0–15) in hospital-based studies (n=48). No tools provided prepandemic mortality risks, and only one presented risks separately across high-risk conditions.18
Cumulative number of COVID-19 risk prediction tools without and with public involvement, number of global deaths, and iterations of patient and public engagement, January 2020–February 2021.
Generation and presentation of mortality risk information
Until prototype release in May 2020, we presented patients with six of the most common moderate-risk conditions (severe obesity, diabetes, cardiovascular disease (CVD), chronic kidney disease (CKD), chronic obstructive pulmonary disease (COPD) and prescribed corticosteroids).20 Table 1 shows the study population for the public OurRisk.CoV version.31 Supplementary material S2 shows conditions and baseline 1-year mortality in prototype and public versions. Comments about limited range of conditions and omission of major conditions in government guidance led to incorporation of all ‘moderate-risk’ (n=67) and ‘high-risk’ conditions (n=20) with absolute all-cause mortality risk. Only 10 conditions had specific RRs usable in analyses (eg mild asthma); for others, we used composite RRs (eg ‘other neurological disease’ for Parkinson's disease; ‘chronic heart disease’ for heart failure) (supplementary material S3). Except mild and severe asthma, there were no COVID-19-specific RRs for disease severity. We therefore provided a ‘slider’ to allow changing of RRs in discussion with specialists, and as new data and research became available. Based on PPIE feedback, to combine tailored risk estimates with as simple an interface as possible, we provided information based in ‘five clicks’ (age, sex, underlying condition, multimorbidity and ‘calculate’).
Study population by moderate- and high-risk characteristics
Patient and public involvement and engagement
Our research began with questions from a patient to a cardiologist (AB) on 17 March 2020: ‘Why am I on the government's list of high-risk conditions for COVID-19? How much does my heart failure and age put me at increased risk of dying? How has my risk changed during the pandemic?’ In two phases of PPIE, pre- (supplementary material S4–S6) and post- (supplementary material S7–S12) prototype release, we identified three themes: information needs of patients,32 usability of the information,33 and shared decision making with healthcare professionals.30,34 In supplementary material S13, we highlight comments in these three themes from the five different types of PPIE in the two phases, with resulting actions. We provide all feedback in full in supplementary material S4–S12.
Informing prototype (March–May 2020)
Information needs
Patients wanted simple data representations, but inclusion of a wide range of conditions and multimorbidity. We decided on initially providing tailored mortality information for a small number of common conditions, with potential to vary assumptions in the model. Given patient concerns about data use and privacy, we did not collect data from users of the tool.
Usability for patients
There were reservations about usability, with a need for further dialogue to make the tool more useful to patients and the public. We therefore focused our prototype on researchers and policymakers (although it would be open to the public), with plans for increased PPIE, feedback and user testing before public release.
Shared decision making
Patients expressed interest in using mortality risk data in discussions with health professionals. However, there were concerns about how this information would link with evolving COVID-19 advice and potential unintended consequences, where risk might be low.
Informing subsequent development of public-facing version (May 2020–November 2020)
Information needs
The prototype calculator with six conditions was released on 12 May 2020, receiving 340,000 unique users (including UK: 63.4%, US: 20.1%, Australia: 4.0%, Canada: 2.8%, India: 1.9% and Germany: 1.1%) within 24 hours of release (1.3 million page views over 1 month) and 136/147 (92.5%) of user feedback emails were positive. Throughout PPIE, there was expression of public need for and interest in mortality risk information for as many of the moderate- and high-risk conditions as possible, ideally curated and maintained within the NHS using NHS data.
Usability for patients
To meet the information needs of patients, wider and more representative PPIE would be necessary and exact use cases would need to be better understood and articulated (supplementary material S7).
Shared decision making
There was greater interest in exploring how mortality risk information might inform shared decision-making in the tool with 87 conditions (compared to the version with only six conditions). However, patients highlighted need for better data (on disease subtypes and severity) and for working with specialists involved in care of each of the 87 conditions to develop the tool.
88.2% (30/34) of the PPIE panel answered ‘agree’ to ‘I think that the NHS should provide patients with accurate risk information, which takes account of disease severity and other factors considered important by my specialist’, and 91.2% (31/34) that patient groups and charities should convene with health professionals to understand and meet patient demands for better risk information. 38.2% (13/34) would consider using the information in conversations with clinicians, or decision making during the pandemic. Despite iterative and responsive changes to OurRisk.CoV, 29.4% (10/34) agreed that it was ready for public use in September 2020 (supplementary material S10 and S11). The panel was unanimous in support of peer-reviewed public reporting of the process to stimulate further development of public-facing risk tools.
Discussion
Key advances
We have three new findings. First, we show lack of public and patient engagement in COVID-19 risk tools to-date and lack of mortality risk information designed for patients with underlying conditions. Second, there was sustained patient and public interest and engagement in developing such information during and beyond the pandemic. Third, we demonstrate feasibility and utility of a single online portal for mortality information for a wide range of conditions, informed by patients and public, giving context for decision making and discussions with health professionals, family members and carers.
PPIE missing from prognosis research
Lack of PPIE in COVID-19 mortality risk tools exemplifies a wider issue in prognosis research. Despite long-standing calls for PPIE in prognosis research,13 we are not aware of any mortality risk tools developed with the public, or a primary audience of patients and public. Knowledge generation on mortality risk has been by, and for, clinicians (or researchers).35 Moreover, although clinicians may make risk tools open to the public, these tools have had limited, if any, PPIE during development36 and use by patients and public is unknown. Recent calls for knowledge coproduction37 should include generation and use of risk data.
Information needs of patients with different diseases
Our research prototype (with only six conditions), receiving 1.3 million page views over 3 months, suggests patient need for mortality risk information. We found only one published survey of people with a long-term condition about their risk information needs. Among 3,000 people with multiple sclerosis, 92% were ‘interested in using tools that generated personalised predictions’.38 People with most of the 87 underlying conditions, which we report, have not been included in quantitative or qualitative research to understand their risk information needs.
Usability of patient information
Patient groups and charities covering the 87 conditions provide some public risk information,39–42 but none provide prepandemic mortality risk by age, sex and multimorbidity, which we provide in a single online portal. We found strong patient support for reporting risks by condition and developing more information on multimorbidity (currently risks for ≥1 conditions and prevalence for top 10 comorbid conditions). Patients with multimorbidity prioritise remaining alive, being independent, and relief from pain and symptoms. Although all-cause mortality is relevant across diseases,11,43,44 accurately measured, and unbiased by COVID-19 testing strategy, admission policies, or cause of death certification, we only tackled the first of these concerns. Patients and public were positive about our 1-year perspective, which is more relevant in the context of ‘long COVID’, but were also interested in the pandemic's indirect effects, whether medical, social, or economic. We updated RRs with current best estimates18,19,27 for each condition: these require updating as the pandemic evolves and as new condition-specific estimates become available. Further testing and development of such mortality risk information with PPIE is needed to improve usability.
Shared decision making
We show that some patients would discuss their risks with clinicians.45 To answer patient questions, clinicians have varying access to data-driven risk estimates, depending on condition. Diseases differ by 90-day COVID-19 mortality,18,27 and potentially direct and indirect effects.46,47 For example, for diabetes, nationwide linked registry data have been used to report RR for COVID-19 mortality,48 but such data are either not reported or not available to patients with many at-risk conditions. Provision of patient-centred risk data is one part of shared decision making, a complex intervention with multiple components, including clinician and patient training, and culture change,49 which requires further PPIE.
New risk information across clinical specialities
We found that patients and public were surprised that an agreed national approach to defining overall mortality risk for people with each of the underlying conditions was lacking. Patients expressed a need for more specific risk information (relative and absolute) tailored to severity of disease, and for more specific risks in people with coexisting conditions.50,51 Disease severity information, such as ejection fraction (heart failure), troponin rise (acute myocardial infarction), disease stage (cancer) and imaging results (many diseases), is often not present in primary care records and exists, for example, in more than 100 disease-specific national quality registers, for cancer,47 CVD46 and other diseases,48 which tend not to include information on multimorbidity, particularly ‘extra-speciality’ multimorbidity. Meeting patient needs for risk information requires new data integration across specialties, with primary care, underlined by the fact that majority of people with CVD or cancer have other conditions.47 The HDR UK Gateway, an open catalogue of existing health data, could be developed for the public to see where data relevant to age and condition exist and could be used.52 Nationwide cohorts (eg 54 million population in England) have become research accessible in 2021,53 and such scale may offer more accurate and useful risk information (Box 1).
Examples of hypothetical patient-led conversations with clinicians informed by OurRisk.CoV
Strengths and limitations
We used large-scale, nationally representative data to produce mortality estimates and were able to gain conduct substantive PPIE in real-time during the pandemic. Our adaptive methodology allowed incorporation of several rounds of patients and public feedback. We included all comments and feedback from PPIE and acted upon it. There are limitations. Our PPIE interactions were virtual, where ‘face-to-face’ may have provided richer information. We established a panel from HDR UK early in the pandemic, which may not be diverse. We began engaging the panel in March 2020, before it was known that minority ethnic groups have higher risk of infection, severe infection and death with COVID-19 (the panel had two ethnic minority members). Responding to immediate needs pre-lockdown, our approach was formative and iterative, without peer-reviewed funding for qualitative research.
Research and development implications
We offer indications for further research into how the tool might be developed for wider public use, presenting (in supplementary material S4–S10) all feedback from patients – positive and negative – which is uncommon in previous PPIE research. First, we suggest the need to better understand the route ‘from patients to patients’ in developing useful risk information tools. We were surprised to find that although only 29% of patients and public felt OurRisk.CoV was ready for public use, the patient panel was clear that the next step was to expose the process to public scrutiny. Second, following previous recommendations,13 there is a need to establish metrics for PPIE methods and outcomes on conduct and reporting which journals and peer reviewers can use. Third, the pandemic offers an opportunity to expand the role of PPIE in clinical practice and research.54–57 Fourth, as more large-scale data analysis from the COVID context becomes possible, it will be important to include further tailored information, so that such tools can be used in different circumstances such as risk assessment for healthcare staff.
Policy implications
Our findings have potential policy implications. First, policymakers need to establish means of answering public questions on risk, which is not currently the responsibility of any organisation (in NHS, third or private sectors). Our findings suggest that multiple stakeholders, methods and ongoing PPIE are prerequisite, perhaps requiring a dedicated organisation (such as the Institute for Health Metrics and Evaluation for the Global Burden of Disease Study). Doctors are required to provide ‘all material risks’ when consenting patients (Montgomery ruling since November 2020),58 underscoring the need to find new ways to generate and communicate risk information. Second, there is a role for charities, patient organisations and patients to collaborate and articulate a framework for better risk information across disease silos.
Conclusion
We provide a mortality risk calculator, informed by patients and public, for 87 underlying conditions in the COVID-19 context. OurRisk.CoV provides risk information for patients to inform decision making and discussions with heathcare professionals, family members, and others during the pandemic. There is an urgent need to better understand what risk information patients and the public want during and beyond the pandemic.
Summary
Supplementary material
Additional supplementary material may be found in the online version of this article at www.rcpjournals.org/clinmedicine:
S1 – Systematic review of 54 COVID-19 risk prediction tools in the general population and in hospitalised patients.
S2 – Forest plots for initial list of conditions (22 March 2020) and for all moderate and high risk conditions for 1-year baseline mortality.
S3 – Underlying conditions.
S4 – Initial questionnaire for PPIE panel.
S5 – Phase 1 public and patient involvement and engagement.
S6 – Selected comments from patients, public and colleagues.
S7 – Selected comments from email, media coverage and social media.
S8 – Phase 2 public and patient involvement and engagement.
S9 – Summary of Phase 3 and 4 public and patient involvement and engagement.
S10 – Focus group discussion recommendations.
S11 – Recommendations of PPI for use of risk information and next steps.
S12 – Summary of Specialist Engagement.
S13 – Thematic analysis of public and patient involvement and engagement in design and development of a public-facing mortality risk calculator.
Conflicts of interest
All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: research funding from Astra-Zeneca to Amitava Banerjee unrelated to the current work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.
Funding
Amitava Banerjee is supported by research funding from NIHR (NIHR200937), British Medical Association (TP Gunton Award), Astra-Zeneca and UK Research and Innovation. Bryan Williams and Harry Hemingway are National Institute for Health Research (NIHR) Senior Investigators and are funded by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. Harry Hemingway's work is supported by: 1. Health Data Research UK (grant number LOND1), which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and Wellcome Trust. 2. Amitava Banerjee and Harry Hemingway are part of the BigData@Heart Consortium, funded by the Innovative Medicines Initiative-2 Joint Undertaking under grant agreement number 116074. This Joint Undertaking receives support from the European Union's Horizon 2020 research and innovation programme and EFPIA; it is chaired, by DE Grobbee and SD Anker, partnering with 20 academic and industry partners and ESC. Alvina G Lai is supported by funding from the Wellcome Trust, National Institute for Health Research (NIHR) University College London Hospitals, NIHR Great Ormond Street Hospital Biomedical Research Centres and HDR UK (Better Care Catalyst Award). Michail Katsoulis is funded by the British Heart Foundation (grant number FS/18/5/33319).
Acknowledgements
A huge thank you to the many members of the Health Data Research UK COVID-19 Patient and Public Involvement and Engagement Panel. The following acknowledges those who were happy to be included this paper but does not represent all members and does not reflect level of contribution: Ian Atkinson, Angela Coulter, Martin Cox, Chris Cuninghame, Marion Cumbers, Rob Dalziel, Rosanna Fennessy, Rebecca Harmston, Ben Johnson, Kevin Knowles, Dianna Moylan, Lizanne Brown, Shilpa Patel, Kelvin Pitman, Amanda Roberts, John Roberts, Bob Ruane, Katie Ruane, Roxane Stirling, Mike Thorpe, Ken Wilkinson, Nickie Witham, Virginie Damoiseaux, Alex Brownrigg, Stella Macpherson, Sarah Markham.
This study is based in part on data from the Clinical Practice Research Datalink obtained under licence from the UK Medicines and Healthcare products Regulatory Agency. The data are provided by patients and collected by the NHS as part of their care and support. The interpretation and conclusions contained in this study are those of the author/s alone. Copyright © (2020), re-used with the permission of The Health & Social Care Information Centre. All rights reserved.
Natalie Banner (Understanding Patient Data), Jonathan Sterne, George Davey-Smith (University of Bristol), Chris Sinclair (Global Initiative) and Gareth Nixon (Global Initiative) provided helpful comments on the calculator and the manuscript.
- © Royal College of Physicians 2021. All rights reserved.
References
Jump to section
Related Articles
Cited By...
- No citing articles found.