ABSTRACT
Background and rationale Reliable prediction of discharge destination in acute stroke informs discharge planning and can determine the expectations of patients and carers. There is no existing model that does this using routinely collected indices of pre-morbid disability and stroke severity.
Methods Age, gender, pre-morbid modified Rankin Scale (mRS) and National Institutes of Health Stroke Scale (NIHSS) were gathered prospectively on an acute stroke unit from 1,142 consecutive patients. A multiclass random forest classifier was used to train and validate a model to predict discharge destination.
Results Used alone, the mRS is the strongest predictor of discharge destination. The NIHSS is only predictive when combined with our other variables. The accuracy of the final model was 70.4% overall with a positive predictive value (PPV) and sensitivity of 0.88 and 0.78 for home as the destination, 0.68 and 0.88 for continued inpatient care, 0.7 and 0.53 for community hospital, and 0.5 and 0.18 for death, respectively.
Conclusion Pre-stroke disability rather than stroke severity is the strongest predictor of discharge destination, but in combination with other routinely collected data, both can be used as an adjunct by the multidisciplinary team to predict discharge destination in patients with acute stroke.
Introduction
There are over 1 million people living with stroke in the UK.1 This number is set to rise to over 2 million by 2035.2 This comes at a great financial cost to health services, social services and carers.2,3 Twenty-five to 28% of these patients are frail, while 51% are considered ‘pre-frail’, approximately double the incidence observed in the general inpatient population.4–6 Frailty is a predictor of mortality, length of stay and functional recovery in the period following an acute stroke.5,7–9
Twelve to 15% of stroke patients will die during their presenting admission, although this number is falling in the UK.1 The survivors are discharged home, to community hospitals or specialist inpatient rehabilitation. Discharge destination from an acute setting is an important predictor of 3-month outcome, and early designation would inform planning and the appropriate allocation of clinical resources.10,11 However, the varied presentations of cerebrovascular disease combined with a high incidence of common comorbidities complicates the early and accurate prediction of discharge destination.
Earlier specification of an appropriate discharge destination leads to better outcomes. Delay in discharge increases cost and is associated with increased mortality, independent of medical specialty.12 In stroke, early transfer for inpatient rehabilitation is beneficial to patients.13,14 Discharge home may be delayed by inadequate planning of community support, where sufficient lead time can be an important factor.15 Additionally, discharge to an inappropriate setting, or inadequate planning, leads to avoidable readmission and morbidity.16–18
Pre-stroke functional performance is a predictor of stroke outcomes.5,7,8,19 We know of a single study exploring the relationship between pre-stroke disability and discharge destination from the acute stroke unit.20 However, this is not validated against an independent dataset. In this regard, stroke medicine is behind other specialties; these data have been used effectively in gastrointestinal surgery to predict discharge destination following resection of malignancy.21
The modified Rankin Scale (mRS) is a recognised measure of functional disability and it has a strong correlation with the Rockwood frailty index.22 The mRS and the National Institutes of Health Stroke Scale (NIHSS) are recorded for all stroke patients presenting in England, Wales and Northern Ireland as part of the Stroke Sentinel National Audit Programme (SSNAP). Despite their widespread use and availability, these data have never been used to predict discharge destination.
Machine learning has shown considerable promise as a tool to inform decision-making in clinical practice, particularly where large amounts of data are available.23–25 However, there are still significant concerns among clinicians regarding implementing tools based on ‘black-box’ learning without an understanding of their rationale.
This study was designed to outline the relationship between pre-stroke disability, stroke severity and discharge destination in order to develop a predictive model for discharge destination that clinicians can understand and trust.
Methods
Data was collected prospectively, for SSNAP, from 1,142 consecutive patients who were admitted to the acute stroke unit at the University Hospital of Wales between 01 January 2015 and 31 December 2016.
Ethics approval was not required for this study as per the NHS Human Research Authority guidelines for Wales and consent was not sought for analysis of an existing, anonymised database.
Stroke severity (baseline NIHSS), pre-stroke disability (pre-morbid mRS) and patient demographics were collected in the emergency department by a health practitioner with specialist stroke training.
Discharge destinations were defined as home, community hospital (an inpatient unit not specialising in stroke), inpatient rehabilitation (defined as an inpatient unit providing specialist stroke care) and death (while on the acute stroke unit). Patients discharged home included those who had an ‘early-supported discharge’.26
Kruskal–Wallis tests were used to assess whether each predictor had an independent relationship with discharge destination. The independence of this association was expressed as a p value (p<0.05 considered significant) and an H statistic was used to express the strength of the correlation.
The data was used to form a training set (n=1,016) and a test set (n=115). These were matched for age, gender, mRS and NIHSS (Table 1). Models derived from the training set were validated against the test set to calculate the accuracy of the predicted discharge destination.
Baseline demographics of training and test groups
A variety of random forest models were trained to predict patient destination from age, gender, mRS and NIHSS. These were tested against a dummy model (predicting the most common destination for all patients ie rehabilitation hospital) using 100-fold cross-validation and performing t-tests on the resulting distribution of test accuracies. A p<0.05 was considered statistically significant. To avoid over-fitting of the model to the test set, hyperparameters were optimised by cross-validation of a number of models within the training set. We then reported the performance of the best model resulting from this procedure against the test set.
For each outcome, we computed the positive predictive value (PPV; the proportion of patients who were predicted to have that outcome actually did have the outcome (known as ‘precision’ in machine learning)) and the sensitivity (the proportion of patients who really did have that outcome were predicted to have it by the model (known as ‘recall’ in machine learning)).
Analyses were performed in Python, using scikit-learn for model specification and training, and eli5 for model analysis.
Results
Of the 1,142 patients, 171 (15.1%) were discharged home, 299 (26.4%) to a community hospital, 555 (49.1%) to a specialist inpatient rehabilitation setting and 106 (9.4%) died. Eleven (1%) of these patients were discharged directly to care homes, and were excluded from this analysis. Median length of stay on the acute stroke unit was 5.9 days (interquartile range 2.9–10.9).
Fig 1 shows the relationship between each predictor and the possible discharge destinations. High NIHSS scores were associated with inpatient rehabilitation or death (H=390.75; p<0.0001). The mRS showed a striking, if complex, relationship with outcome (H=356.4; p<0.0001); low and high mRS scores were associated, respectively, with discharge home and to inpatient rehabilitation. Patients who returned home had a lower mRS, while those who became inpatients had the highest. Community hospital and death had intermediate values; age and gender had no clear relationship with outcome.
Relationship between predictors and outcome. a) National Institutes of Health Stroke Scale (NIHSS). b) Age. c) Modified Rankin Scale (mRS). d) Gender.
A variety of machine learning models were used to predict discharge destination. Initial exploration showed that random forest models provided the best performance on the validation set (data not shown). All subsequent results are for random forest models with different combinations of input variables.
The results are shown in Fig 2. The dummy model (grey), which predicts the most common class for every patient (Fig 2a), achieved an accuracy of 50%. This is because 50% of patients are discharged to inpatient rehabilitation. For models utilising only a single predictor (blue), mRS was the strongest (p<0.001 vs dummy model), while NIHSS and age alone were no better than the dummy model (NIHSS: p=0.86; Age: p=0.84).
a) Performance of trial models plotted against dummy model achieving roughly 50% accuracy. b) A confusion matrix for the best performing model, applied to the test set. This illustrates when the model was correct (diagonal entries) and which categories it tends to confuse (off-diagonal entries). Note that the confusion matrix reflects the ordering of the outcomes; patients who are discharged to community hospital are never assigned to the death category, and vice versa. Numbers are a fraction of patients for each true outcome, so rows sum to 1. mRS = Modified Rankin Scale; NIHSS = National Institutes of Health Stroke Scale.
Combining pairs (purple) or triplets (green) of predictors improved performance. The best performing pair was mRS and NIHSS. All four predictors together (teal) performed significantly better than any other model (p=0.021 to 2.1 × 10-26), achieving 70.4% on the test set.
We analysed the performance of our model on the test using a confusion matrix (Fig 2b). Categories more similar in severity (eg home and community hospital) are confused more often than very stark outcomes.
The destination values for death were PPV 0.5 and sensitivity 0.18; for community hospital, PPV 0.70 and sensitivity 0.53; for inpatient care, PPV 0.68 and sensitivity 088; and for home, PPV 0.88 and sensitivity 0.78.
An intuitive understanding of the decisions such a model makes is difficult in a static setting. We have deployed an interactive version of this analysis online (https://stroke-discharge.herokuapp.com) and invite the reader to explore the effect of different input values upon the models’ outputs. This allows the user to visualise where an individual patient lies in relation to the training data, and thus how much confidence to place in the predictions made by the model for that patient.
Discussion
This collaboration between data scientists and clinicians used pre-stroke disability and combined it with stroke severity on admission, age and gender to train a random forest model to predict discharge destination from the acute stroke ward. It is the first predictive model of which we are aware that has been validated against an independent dataset, and, in this dataset, predicted discharge destination across all groups with an accuracy of 70.4%.
Pre-stroke disability, expressed by the mRS, is the strongest single predictor of discharge destination when incorporated into a random forest model. Stroke severity is correlated with certain discharge destinations. However, in a patient cohort where nearly 50% are discharged to inpatient rehabilitation, this was a sufficient ‘signal’ to provide useful prediction when compared with a dummy model predicting the most common discharge destination for every patient. In essence, stroke severity alone is not sufficient to determine discharge destination. However, it can considerably enhance the performance of a model that incorporates pre-morbid baseline.
This model highlights the important difference between pre-stroke disability (and all that it implies about a lack of reserves, vulnerability and a predisposition to decompensation) and the current concept of stroke severity. The NIHSS is computed by combining scores describing the severity of relatively focal deficits (such as weakness of a limb, loss of a visual field or sensory loss) leading to a score summated from 13 ordered categorical scales. It does not include tests of functions (such as walking, swallowing or coughing) and, from it, in a patient who is conscious, very little can be inferred about their level of function. A 20-year-old athlete with a dense hemiparesis scores the same for a given constellation of deficits as a 90-year-old with same, regardless of any differences in dependence or pre-morbid disability.
Incorporating age into the model alongside measures of pre-stroke disability (mRS) and stroke severity (NIHSS) improved its accuracy, which is supported by other studies which have explored the relationship between biological age and stroke mortality.27
Our model suggests that pre-stroke disability and stroke severity are most effective at identifying patients who are discharged home and to inpatient rehabilitation and was less effective in predicting death or transfer to a community hospital. Arguably, discharge to home or to a stroke rehabilitation unit are the most important in clinical practice. Early discharge planning for home discharge helps to avoid delay in assessment, the organisation of care and the provision of home equipment or adjustments; all sources of obstruction to prompt discharge.28 Early identification of the patients who will benefit most from intense inpatient rehabilitation is important to optimise functional recovery, and may reduce length of stay in the rehabilitation setting.13,14,29 Our model was less effective in identifying patients who died during their first admission or those discharged to community hospital; the latter may reflect the fact that discharge to community hospital is often dependent on situational and contextual factors (eg availability of beds, family dynamics etc) rather than a particular level of disability.
The mRS was not developed as an index of frailty, although it correlates well with established indices of frailty.4,7,22 Thus, this study relates closely to the work of Seamon et al.20 They took a large cohort of Medicare patients in the USA to explore the influence of frailty (using the Faurot frailty index) and stroke severity on discharge to an inpatient rehabilitation setting. Their logistic regression analysis found that non-frail and pre-frail patients, and frail patients with low stroke severity scores, were more likely to be discharged to inpatient rehabilitation. This is at odds with our findings, but may reflect a different model of care in the UK. Nearly 50% of the patients in our study were discharged to inpatient rehabilitation; this is only true for 22% of their cohort. Their methods were substantially different to those found in this paper. They did not incorporate gender or age in their calculations, which were both shown to increase the accuracy of the model in this study. In addition, logistic regression assumes that there is a linear relationship between variables; our model is not limited by this assumption.
Clearly, the high values for discharge to further inpatient care or to home are of direct relevance to a multidisciplinary team discharge planning meeting. In addition to the clinical observations and knowledge of the team, the model provides a PPV of 0.88 for the patient going home or of 0.68 for the patient going on to further inpatient rehabilitation. These predictions provide additional information which may inform the discharge planning process, and enhance the appreciation within the team of the importance of pre-morbid disability as distinct from stroke ‘severity’, as currently described.
Evidently, the simplicity of this model is a strength. However, it cannot account for the wide range of situational and social determinants of discharge destination. Nevertheless, we hope this model will be a useful adjunct to the stroke team, and help to inform their approach to navigating these contextual factors.
This model used each patient’s first discharge destination and cannot be used to predict mortality following discharge from the acute setting; clearly this is a limitation of the model, due to the large numbers of deaths that occur after patients are discharged from the acute ward. Another limitation is that this model does not distinguish those discharged home with care packages and home-based therapy from those who are more independent. These subdivisions within the home discharge cohort are obviously very important, particularly in relation to the orchestration and choreography required for each discharge.
The reliability of the NIHSS has been called into question. It is prone to grading some of the most disabling consequences of cerebrovascular diseases (eg lateral medullary syndrome, cortical blindness and aphasia) as minor or moderate. This is clearly an issue for a significant minority of profoundly affected patients.30 Nevertheless, it is a widely accepted and standardised scale that is collected for all stroke patients in the UK within 24 hours of admission.
Conclusion
This model, comprising data available at admission, highlights the importance of appreciating the difference between disability and stroke severity. Made available to clinicians for the first time, it has the potential to be a useful adjunct to the discharge planning process in the context of a multidisciplinary team. We hope that this research will stimulate further work between data scientists and clinicians, and improve clinician confidence in clinical modelling. In addition, models such as this may have significant implications on workforce planning and resource distribution.
Summary
Supplementary material
Additional supplementary material may be found in the online version of this article at www.rcpjournals.org/clinmedicine:
S1 – Data science terminology for the clinician.
Acknowledgements
Thanks to the whole multidisciplinary team on the acute stroke ward, University Hospital Wales.
- © Royal College of Physicians 2021. All rights reserved.
