Abstract
The term ‘Black Wednesday’ has been used to describe the August national changeover day, a day when a new cohort of inexperienced doctors start work, many of whom are absent from patient care to attend organisational induction and mandatory training. In this paper, we report on the development and implementation of a novel, interactive e-learning programme for induction and mandatory training for junior doctors in a district general hospital in south-west England from August 2013. This comprehensive mandatory-training programme with summative assessment saved 19.5 hours of trust time per trainee. Since the programme’s inception, the completion rate has been 100% (n = 370). Subgroup analysis of starters from August 2013 (n = 141) showed that 85.7% completed by day 1 (mean time of completion 3.0 days before day 1, standard deviation 14.2 days). Importantly, 90 minutes of induction was freed on Black Wednesday, enabling earlier, ward-based clinical orientation, thereby enhancing patient safety. We believe that this is the first programme to combine induction with fully assessed, comprehensive mandatory training in a single package. Such an approach is suitable for widespread application and is to be implemented regionally.
Background
In 2009 Dr Foster Intelligence published a report showing a rise in mortality rates during the first week of August.1 This led to the British press coining the expression ‘Black Wednesday’, describing the first Wednesday of August as the ‘NHS Killing Season’.2,3 The study examined 300,000 emergency admissions over 9 years and showed a 6–8% rise in mortality during the first week of August compared with the last week in July.1,4 This rise was attributed to the nationwide influx of new junior doctors. Subsequently, attention was focused on junior doctors’ preparedness to practice, including shadowing schemes and the provision of effective induction and mandatory training (MT).
A study from the University of Bristol suggested that the introduction of a shadowing scheme led to a 52% reduction in errors made by FY1 doctors.5 Sir Bruce Keogh announced in 2012 that shadowing would become a national requirement.6 However, less emphasis has been given to induction and MT and despite recommendations from the Academy of Medical Royal Colleges (AoMRC),7 completion rates remain poor.8
The General Medical Council states that comprehensive induction and mandatory training should be delivered to all trainees in a time-efficient manner to allow sufficient time for ward-based shadowing.9 However, the provision of effective induction and MT for doctors is challenging. Annual August trust induction requires many doctors to be absent from patient care for a significant period, with implications for patient safety. Nationally, the NHS Litigation Authority, NHS Protect and the Health and Safety Executive set standards for MT, but these are rarely achieved. The AoMRC has also produced a set of recommendations for safer trainee handover,7 in which it acknowledges the increasing body of evidence to suggest that simultaneous trainee changeover is associated with adverse patient outcomes and reduced trainee satisfaction.
One of the four areas of recommendation from the AoMRC focuses on the provision of high-quality comprehensive induction and MT. The report supports the use of novel strategies to deliver induction and MT in a time-efficient manner, minimising the time that doctors spend away from clinical care. In our trust (the Royal United Hospitals Bath NHS Foundation Trust) in 2012, it was estimated that 29.5 hours were required for trainees to complete trust induction and all MT elements (Table 1).
Delivery of trust induction and mandatory training 2012.
Ten MT topics were covered at the face-to-face trust induction, each as a 5-minute talk. These topics had to be followed up with an e-module or face-to-face session to meet the MT requirement and provide verification of learning. In practice, very few trainees completed the follow-up modules. Additionally, there were issues around recording completion, because modules were undertaken at different times by different departments. As a result, no junior doctor within our trust had been recorded as completing all elements of induction and MT in 2012 or before, constituting a significant safety risk. This project was designed to address this issue, with the aim of achieving close to 100% compliance with MT by day 1.
Methods
We developed a comprehensive, web-based electronic induction (e-induction) and electronic MT (e-MT) package to be completed by all trainees and relevant non-training grade (NTG) doctors before starting in the trust. This provided doctors with trust induction and full MT compliance, with verification of learning via a multiple-choice summative assessment.
We created a working group of multidisciplinary professionals, both medical and non-medical, to form the Dynamic e-induction team. The team worked closely with subject matter experts from corresponding departments within the trust to agree standards for each topic of e-MT, shape the content of each module and develop assessments. We liaised with the development company, who tailored the package to our needs.
Many regions have already successfully implemented online e-induction. However, to the best of our knowledge no other trust had incorporated comprehensive MT and a summative assessment of learning. Therefore, we integrated 13 e-MT components into a modified, interactive, multimedia, scenario-based e-induction programme based around video footage of individual patient scenarios.
We used a Moodle-based platform, a open-source software learning management system, to incorporate multiple-choice-based summative assessments covering 13 MT elements:
infection control
conflict resolution
equality and diversity
fire safety
safeguarding adults
safeguarding children
manual handling
blood transfusion
information governance
medicines management
venous thromboembolism
health records
death certification.
The package was highly interactive and required users to work through a series of six cases, contextualised in the setting of an emergency department during a major incident. Information from each of the 13 MT elements was integrated throughout the six cases and interspersed with relevant assessment questions. Additionally, each user started with 100 points and lost a point for each incorrect answer, which facilitated engagement with the package. By setting the scene during a major incident, the themes of major incident procedures and triaging skills were introduced. All trainees were invited to complete the e-induction by email 2 weeks before their start date. A compensatory lieu day was granted to those who completed the e-induction before their first day of work.
The 13 MT topics were consolidated into 10 multiple-choice-based assessments using Angoff pass-mark settings10 to identify minimum competency. If a trainee failed an assessment, they were given the opportunity to retake and redirected to the relevant course material facilitated by a dropdown subject index. Trainees had to pass all assessments before completing a final evaluation consisting of combination of Likert scales and free text. This feedback was used to direct subsequent improvements to the package. A certificate of completion was then issued. The postgraduate medical centre logged completion of e-induction and e-MT and this was recorded on the trust managed-learning environment database. To demonstrate an improvement in both induction and MT, we captured data relating to timelines of completion, comprehensiveness of training, assessment outcomes and programme evaluation.
Results
The package was introduced in August 2013, and resulted in an improvement from no trainees completing all elements of trust induction and mandatory training to all 370 trainees completing them 2 months after starting by August 2014 (Fig 1). This finding shows the difficulties in capturing all August starters due to shift work and night duty, indicating the inherent problem of face-to-face group training in an acute hospital. Additionally, a system relying on mass induction in August fails to deal with those joining during the other 11 months, notably those not in training posts.
Completion rates for mandatory training, 2012–14. Yellow bars represent completion of full mandatory training by topic (e-modules and/or face-to-face) in 2012 (n = 207). Red bars represent those who attended August face-to-face induction in 2013, with brief 5-minute introductory talks on 10 mandatory topics, without assessment of learning (n = 141). Green bars represent completion of e-MT by all starters in the period August 2013–July 2014 (n = 226). Blue bars represent completion of e-MT by August 2014 starters (n = 144). e-MT = electronic mandatory training; VTE = venous thromboembolism.
We analysed a subgroup of August 2013 starters to compare times to completion and pass rates. In total, 97% of trainees completed the package by the end of August 2013, with 130 trainees (85.7%) completing the induction before day 1 in August 2013 (mean time to completion 3.0 days before day 1, standard deviation 14.2 days). Time to completion varied among different grade of doctor, with the longest times noted in higher specialty trainees (ST3 and above) and NTG doctors. Thirty days after the initial invitation to undertake the e-induction, five trainees (3%), all of whom were ST3 or above, had not completed the package. All trainees completed the package by 19 September 2013.
An overall analysis of times to completion from August 2013 to end of August 2014 is shown in Fig 2. Overall, FY1 doctors were the most timely at completing the package (mean completion 8.9 days before day 1). The ST3+ group took the longest (mean completion 2.8 days after day 1), with a much wider scatter (standard deviation 15.7 days), and needed the most prompting to achieve 100% compliance. There was no difference in the proportion of timely completions between FY1s and FY2s, or between FY1s and those on the GP training scheme. However, core medical trainees, NTG doctors and ST3+ doctors were much less likely than FY1s to complete the package on time and overall took much longer to complete it.
Time to completion of e-induction and e-mandatory trainings package by grade of doctor. CMT = core medical training; FY = foundation year; GPST = general practice specialty training; NTG = non-training grade; ST = specialty trainee.
Fig 3 shows a subgroup analysis of pass rates according to the number of attempts made at assessments for MT components for August 2013, according to trainee grade and country of primary medical qualification. A total of 1,534 assessment attempts for all categories of MT were undertaken by August 2013 starters. Doctors more junior than ST3 passed 963 of 1, 066 (90.3%) MT component assessments, whereas those ST3 or above passed 342 of 468 (73.1%). The failure rate for ST3+ doctors was significantly higher than for their junior colleagues (p < 0.001, χ2 test). Doctors whose primary medical qualification was obtained in the UK passed 1,242 of 1,426 (87.1%), whereas those who qualified outside the UK passed 63 of 108 (58.3%) attempts – a highly significant difference (p<0.001, χ2 test).
Comparison of pass rates by grade of trainee and country of primary medical qualification, for all August 2013 starters. ST = specialty trainee.
Feedback from the evaluation forms was positive, particularly among the FY1 doctors, who found it interesting, relevant and interactive, and liked being able to work at their own pace. Qualitatively, users commented that this format provided greater flexibility, keeping users more engaged than face-to-face induction. Quantitatively, the most positive response was that users found the day in lieu a good incentive to complete the package. More junior trainees than senior trainees made positive comments.
Discussion
Scenario-based e-induction packages have been used in several other postgraduate institutional regions. However, to the best of our knowledge, we believe that ours is the first programme to include the innovative addition of key MT with a comprehensive summative assessment tool, which ensured that trainees demonstrated evidence of learning to meet minimal competency. By providing all MT in a single package, we demonstrated dramatic improvements in both completion and recording of MT. This programme had several advantages over our old induction and MT model. Reliance on junior doctors to complete several additional face-to-face modules was previously unsuccessful. Although some limited MT can be provided during face-to-face induction on Black Wednesday, this setup requires front-line junior doctors to be absent from patient care. Around 15% of trainees are usually unable to attend face-to-face induction because of shift work, night duty or leave. Furthermore, only a limited amount of information can be presented without opportunity for robust verification of learning. E-learning provides comprehensive induction with assessed MT and can be completed at the learner's own pace. Furthermore, there is more time for patient care and clinical orientation on day 1. The inclusion of a summative assessment allows some indication of learning and provides information on areas where further learning is required.
Our old programme of induction and MT (a summation of face-to-face induction and MT and additional facilitated e-learning modules) took up 29.5 hours of trust time per trainee. By implementing this comprehensive e-package, the trust time consumed was reduced to just 10 hours per trainee (2 hours of face-to-face induction and compensatory lieu day per trainee). Because trainees were asked to complete the e-package in their own time, the numbers of hours saved by the trust per trainee remained static at 19.5 hours. Data from Dynamic Business Services showed that the mean time taken by trainees to complete e-induction was 4.5 hours (standard deviation 3.1). We do not have data for time taken to complete the summative assessments. However, a reasonable approximation was that, on average, a trainee would have spent 6.5 hours completing this package in their own time.
Importantly, there was a 90-minute reduction in face-to-face hospital induction, which freed up 216 junior doctor hours within the trust on Black Wednesday in 2014 – time that was therefore made available for patient care, thereby improving safety. Even when the 19.5 hours of trust time saved per trainee was rounded down to two working days, it represented a saving of £56,144 of junior doctor hours when using the mid-point salary scale for each grade of doctor for the August intake. The total cost of the package including Moodle functionality was £45,000, which was thus recouped within the first year in terms of additional doctor time. Additional hospital personnel time has also been substantially reduced. Subsequent implementation has allowed a two-thirds cost reduction for other acute hospital trusts in the south-west region. Furthermore, improved training adherence improves the overall hospital MT compliance, with potential financial benefits from improving NHS Litigation Authority compliance.
Feedback was positive, with users rating the flexibility and interactivity of the programme and supporting the concept of a regional training passport. Users commented that there was a friendly and welcoming feel to the package, which put them at ease with joining a new trust. The FY1s in particular felt that, by completing the programme, they were more prepared to start their job within the trust.
It is challenging to devise a package that is concise yet covers material in sufficient depth to meet MT requirements. There are national standards, but there is significant variation in how they are delivered and assessed between different NHS organisations, with different interpretations of the depth and duration of training required, and the balance of face-to-face versus e-learning. Junior doctors who rotate between different organisations during their training could be particularly challenging to engage with, and ensure that their MT is recorded could be difficult. There may be a tension between ensuring 100% uptake of a minimal package with a clear summative assessment and providing a more intense package with poorer uptake rates.
The users completing the package ranged from FY1 to ST8 level – a vast range of experience levels, posing further design challenges. However, although senior trainees might feel that they have already learned this information, it is a national requirement to maintain up-to-date MT, and reaccreditation is required every 12–36 months, depending on the topic. Furthermore, although more senior trainees were less positive in their feedback, it was interesting to note that pass rates for MT were inversely proportional to the seniority of the doctor undertaking the assessment (Fig 3). Our data demonstrated that doctors whose primary medical training had been outside the UK were less likely to pass each component of MT than were those who had trained in the UK. One explanation is that training in a UK medical school might incorporate more complex, culturally based themes relating to MT that are not necessarily covered in non-UK medical schools.
The content of modules does not change significantly over the years, and much remains generic across local education providers. Therefore, our preferred solution is the creation of a regional induction and MT passport allowing transfer of completed training between trusts. Health Education South West launched this project in August 2015, incorporating a regional steering committee to agree content. The aim is to ensure that all trainees remain up to date but without unnecessary duplication when moving between trusts within the region.
There are a number of limitations to be considered. The initial capital outlay was substantial. However, subsequent economies of scale have significantly reduced this cost. We recognise the importance of a tailored approach for more senior trainees, which will be addressed by the Health Education South West Steering Group. We also appreciate that some face-to-face training will always be required, and were mindful to avoid making the package too impersonal. With that in mind, we retained the initial welcome introduction from senior members of the organisation but still aimed to place doctors in their clinical areas by mid-morning.
Conclusion
The creation of a comprehensive e-induction and MT e-learning package led to vastly improved compliance for junior doctors within our trust. The package is cost effective and time efficient. By providing training in this format, with the novel inclusion of e-MT and summative assessment, we have improved doctors’ preparedness to practice. Ultimately, this has led to increased trainee presence on the ward during Black Wednesday, facilitating greater focus on patient care and supporting patient safety.
Acknowledgements
We would like to acknowledge key members of the Royal United Hospital team, including Amanda Gunning from the education centre, Kelly Tillson and Gillie Atkins from the postgraduate medical centre and Richard James and Doug Quiddington from IT. We also thank Julia Gilbert, Ian Crossley, Lisa Welch and James Munroe from Dynamic Business Services, and Prof Selena Gray, deputy postgraduate dean for Health Education South West, for her helpful comments in preparing the manuscript.
- © 2016 Royal College of Physicians
