ABSTRACT
Background In order to provide safe, high-quality hospital care, it is essential that doctors are optimally deployed. Digital platforms for rostering doctors improves both the quality of hospital care and the wellbeing of doctors. We evaluated the usability and acceptability of the HealthRota© e-rostering system.
Methods We circulated an electronic survey among doctors with current/previous experience of using HealthRota, and measured usability (with a system usability score (SUS)) and acceptability. We collected data on how doctors believed HealthRota affected wellbeing, patient safety and training.
Results There were 209 responses (67 non-training clinical fellows, 114 training doctors and 27 consultants) from an estimated 350 doctors who have used HealthRota. Overall, the median SUS score was 86/100 (interquartile range (IQR) 75–97), and there were no differences by role (p=0.118), age (p=0.632) or years of experience (p=0.963). The median Likert score for recommending HealthRota to a colleague was 5/5 (IQR 4–5). There were no differences in role (p=0.477), age (p=0.904) or years of experience (p=0.930). Doctors suggested that HealthRota improves patient safety (4/5 (IQR 3–5)), junior doctor training (4/5 (IQR 3–5)), doctor wellbeing (4/5 (IQR 3–5)) and out-of-hours cover (4/5 (IQR 3–5)).
Conclusion HealthRota is usable and acceptable to doctors and improved the quality of patient care and doctor wellbeing. This serves as evidence to broaden the use of HealthRota.
Introduction
In order to provide safe, high-quality hospital emergency and inpatient care, it is essential to ensure that medical staff are optimally deployed.1 The European working time directive, annual leave, out-of-hours cover, unscheduled time off (including sick leave), vacant training posts, continuing professional development and, more recently, the COVID-19 pandemic need to be taken into consideration when planning hospital doctors' rotas.2 Timetabling doctors to ensure these requirements are met is challenging for many reasons, including ensuring adequate training opportunities.3 A hospital doctor's overall timetable is often a composite of several timetables that encompasses the emergency department or base ward clinical duties, outpatients responsibilities, on-call duties, non-clinical duties (including teaching, medical management and research time) and leave. The status quo in many departments is to keep track of these individual timetables using spreadsheets that are often managed in silos by different departments and members of staff within a single hospital setting. Clearly, this is inefficient and time consuming and can lead to unsafe staffing levels. Furthermore, inadequate rota processes lead to poor morale, reduced wellbeing and health among doctors, and further staffing issues.4,5 The recent COVID-19 pandemic and redeployment of doctors has meant that more robust and flexible hospital-wide systems are required to maintain safe hospital care across specialties and departments.6 There exist many off-the-shelf electronic rostering systems that aim to streamline doctor rotas, however, these systems are rarely able to manage multiple teams (including specialty on-call rotas in addition to acute medicine and the emergency department) with joint on-call commitments, doctors working less than full time and working across multiple hospital sites.1,7 Furthermore, the changes occurring in acute medical training from August 2022 in the UK mean that other medical specialties (genitourinary medicine, palliative medicine, neurology and renal medicine) will be contributing to the acute medical take and, therefore, more integrated hospital-wide rota systems will be needed.
Similar to other settings, locally, we have reduced locum spend by filling vacancies in training posts with non-training (clinical fellow) posts who spend up to 25% of their time teaching in our medical school or participating in research and innovation projects.8 There are examples demonstrating that using digital platforms for rostering doctors improves both the quality of hospital care and the morale and wellbeing of medical staff and, thus, may reduce the requirement for locums.9–11 More integrated and automated medical workforce management is needed to future-proof and maintain safety and quality of medical hospital rotas providing flexibility and bespoke elements for the needs of individuals and teams. Doctors, particularly junior medical staff, value rota systems that are flexible and provide timely notice of duties and the ability to request leave easily.4
HealthRota© is a digital e-rostering system designed for doctors, accessible via an android or IOS application (app), or a desktop version, which creates ‘traditional’ templated rotas but also has the ability to create annualised rotas; it is are being used at the Royal Sussex County Hospital, Brighton, and the Princess Royal Hospital, Haywards Heath. Annualised rotas work in reverse from standard rotas where, rather than needing to request leave, actual clinical hours are calculated and rostered when doctors are available to work, within the confines of what is needed for service. All non-clinical time (such as annual leave, study leave, consultant non-clinical time (sPA) and professional development time) are included in the doctor's time off and only the correct numbers of clinical hours are rostered. Doctors can mark themselves as available or unavailable to work via their HealthRota app, allowing bespoke, flexible rotas to be created. The HealthRota software calculates annualised hours based on individual user contracts and job plans, allowing workforce planning based on actual clinical hours available; importantly, annualised rotas are not depleted by annual leave or study leave requests made once rotas are published. Therefore, rotas can be created that are locum free even after all types of leave have been taken. In addition, clinical fellow posts with 25% non-clinical time can easily be created allowing dedicated time for clinical work and non-clinical work. HealthRota has in-built features to optimise rostering doctors; for example, highlighting rota non-compliance across multiple rotas, in real time, allowing for easy shift swapping and more efficient use of available staff on complex rotas, rather than fixed inflexible templates traditionally used for standard rotas. Other features of HealthRota include shift swapping via the app, requesting leave via app, real-time rota compliance checks in line with contract rules, working across multiple rotas compliantly, self-rostering via the HealthRota app, a locum module that can support the booking of locum shifts directly via the HealthRota app, a template builder that analyses hours and checks rota compliance, and a work schedule and pay document generation based on annualised rota templates or individualised rotas.
The aim of this study was to evaluate the usability (using the system usability scale (SUS)) and acceptability of the HealthRota digital platform among doctors who are current and former users.
Method
We distributed an electronic survey using MS Forms via an email link to doctors working in the trust and those who previously worked in the organisation as part of a medical rota evaluation and needs assessment. We gathered information on the doctor's role (consultant, doctor in training or doctor in non-training post), age in years and years of experience. We measured usability using a modified SUS tool to evaluate the usability of the HealthRota platform.12 SUS is widely regarded as the industry standard for accurately assessing the usability of a digital system due to the robust nature and structural design of the tool, and regarded as the most ‘accurate’ compared with other commonly used usability tools.13 Previous work has shown that a modified SUS tool can be used where appropriate as the tool has enough robustness to sustain mild modification without affecting the overall interpretation.12 The modified SUS comprises nine statements that are directly related to the system being assessed and aimed at users who have experience of using the system within the context for which it is was designed. Each statement within the SUS tool has a five-part Likert scale ranging from ‘strongly disagree’ to ‘strongly agree’. Four statements were selected that were associated with ‘strongly agree’, and five statements were selected that were associated with ‘strongly disagree’. Questions were alternated between ‘strongly agree’ and ‘strongly disagree’ to prevent biases / reduce response acquiescence biases and encourage more careful consideration of each response. Each response has a numerical value assigned based on a complex scoring algorithm, resulting in an overall usability score. As only nine statements out of the 10 are scored, the overall scores range from 0–36 (rather than 0–40). The absolute SUS score can be calculated by multiplying the final score by 2.78 (100/36), instead of 2.5 (100/40).
We measured acceptability by asking doctors if they would recommend HealthRota to a colleague (using Likert score out of 5 where 5 is ‘definitely’ and 1 is ‘not at all’). We also asked doctors (using Likert scores out of 5) how easy they found requesting annual leave and study leave; if they believed HealthRota improved patient safety, junior doctor training, doctor wellbeing and out-of-hours cover; and whether HealthRota was a factor in recommending or accepting a post in our hospital. We stratified the results by age, using four groups broadly representing the age of doctors throughout their careers 24–30, 31–35, 36–40 and over 40 years. We used the Kruskal–Wallis test to compare median SUS scores and Likert scales within 95% confidence intervals. This was a local quality improvement project and, therefore, research ethical approval was not required.
Results
There were 209 responses from an estimated 350 doctors working in the emergency department and medical specialties including acute medicine who had used the HealthRota platform. The respondents identified themselves as either non-training doctors (clinical fellows; n=67), training doctors (n=114) or consultants (n=27). The median age of respondents was 30 years (IQR 27–35) and 46 had 0–2 years of experience, 83 had 3–5 years of experience, 38 had 6–10 years of experience and 41 had >10 years of experience. Overall, the median SUS score was 86 (IQR 75–97): consultants 86 (IQR 39–49), training doctors 92 (IQR 76–97) and non-training doctors 81 (IQR 72–92). There were no differences in SUS score by role (p=0.118), age (p=0.632) or years of experience (p=0.963). The median Likert score for recommending HealthRota to a colleague was 5/5 (IQR 4–5). There were no differences in recommending HealthRota to a colleague by role (p=0.477), age (p=0.904) or years of experience (p=0.930). Respondents scored the ease of requesting annual leave using HealthRota (using a median Likert scale) as 5/5 (IQR 4–5) and the ease of requesting study leave as 5/5 (IQR 4–5). They suggested that HealthRota improves patient safety (4/5 (IQR 3–5)), improves junior doctor training (4/5 (IQR 3–5)), improves doctor wellbeing (4/5 (IQR 3–5)) and improves out-of-hours cover (4/5 (IQR 3–5)). The respondents suggested that HealthRota was a factor in recommending the post to a colleague (3/5 (IQR 3–5)) and HealthRota was a factor in accepting the post (3/5 (IQR 1–4; Table 1).
System usability scale, acceptability and other attributes of the HealthRota© digital platform among doctors
Discussion
We have shown that among the 209 doctors who responded to our survey, the HealthRota platform has a ‘best imaginable’ SUS (86/100) and is highly acceptable (doctors would recommend HealthRota to a colleague (Likert score 5/5)). Furthermore, we found that respondents using the HealthRota platform found it easy to request annual leave and study leave; and suggested HealthRota improves patient safety, doctor wellbeing and out-of-hours cover, and is a factor in recommending a post to a colleague and accepting a post in our hospital.
We believe that this is the first time a digital rota platform for hospital doctors has been evaluated using a modified SUS score. A SUS score of above 86 is described as ‘the best imaginable’ and compares with other digital platforms very favourably (eg PowerPoint scored 74, iPhone scored 78, amazon.com scored 82 and an ATM scored 83).14 Although the SUS score was not intended to be used to assess digital rota platforms for doctors, previous work has shown the score to be resilient when used in diverse settings and can be modified for use in different settings.3,10 It is reassuring to know that HealthRota is usable for the doctors using the platform that will facilitate further development of the platform for other specialties (eg surgery, obstetrics and gynaecology). Furthermore, we have shown that the usability transcends doctors with varying roles, age and experience.
Our data suggest that HealthRota was highly acceptable as doctors would recommend the platform to a colleague and that there were no differences in acceptability by role, age or years of experience. Previous studies have shown that when there is resilience and flexibility in doctors' rotas with inbuilt functionality to organise annual leave and on-call swaps, the rota system is more acceptable. 1 There is no validated tool for assessing acceptability of a digital rota platform and we adapted a patient reported experience measure (PREM) used universally within healthcare to broadly assess the experience and acceptability doctors have of using the platform. Ideally, validated tools should be developed to be able to assess acceptability of digital rostering platforms for doctors.
Given the challenges of managing and maintaining the quality of care of patients in the emergency department and hospital inpatients, it is reassuring to see that the users of HealthRota who responded to our survey believed that HealthRota improved patient safety and out-of-hours cover. Other studies have shown that optimising medical staffing configurations and reducing the need for locum doctors to cover vacancies improved patient care.1 We also found that HeathRota provides flexibility for doctors when booking leave (including study leave) and was believed to improve doctor training and wellbeing. To our knowledge, there haven't been any assessments of rostering systems on the quality of hospital inpatient care to date. There are data showing how ensuring rotas and systems managing rotas affect the wellbeing of the doctors involved.4 Interestingly, doctors responding to the survey suggested that HealthRota was a factor in recommending our hospital as a place to work and, to a lesser extent, accepting a position in our hospital. Furthermore, there are no published data on the implementation and evaluation of other digital health rostering systems to compare with HealthRota; more research is needed focusing on the evaluation, effect on patient care and doctor wellbeing on systems being used.
The strength of our study is that we were able to include over 200 hospital doctors of varying roles and years of experience who have used the HealthRota digital e-rostering platform as part of their working lives. There were several limitations including being a single-centre study without a control group. The survey was managed and circulated by the clinicians who manage the junior doctors' rota rather than an independent researcher, which may have introduced some bias. Due to the rotational nature of most of the doctors using HealthRota, we did not collect information on the current specialty they were working on and were not able to compare different specialties or departments.
Overall, we have shown that HealthRota is usable and acceptable to doctors working in the emergency department and internal medicine, provides flexibility and is perceived as improving the quality of patient care and doctor wellbeing. This serves as evidence to be able to broaden the use of HealthRota to other specialties and grades of doctor in our hospital and beyond.
Conflicts of interest
Rob Galloway originally turned the concept of annualised and self-rostering rotas into a product alongside developers at HealthRota. He is a non-paid consultant to the company but owns a 10% stake in the company.
- © Royal College of Physicians 2022 All rights reserved.
Jump to section
Related Articles
Cited By...
- No citing articles found.