Effects of experimental inductions for newly qualified doctors on competence at clinical procedures
Abstract
Up to 96% of newly qualified doctors fail one or more clinical procedure tests. Their entrance into work in hospitals has been associated with significant reductions in patient safety and an increase in patient mortality. Curriculum changes offer one solution. Another solution is the introduction of clinical skills inductions (orientations) before doctors’ first day at work; the failure rate for one or more clinical tests can be reduced from 96% of new doctors to 27% after just a 5-day experimental induction. Experiments reported in the literature showed improvements in new doctors’ competence at intravenous line insertion and taking blood after a 5-day or 2-week induction, intravenous drug administration after a 5-day induction, certifying death, prescribing and out-of-hours tasks after a 2-week induction, and lumbar puncture and spirometry after a 1-day induction. Examined performance after a 5-day induction also showed improved objective structured clinical examination (OSCE) scores on blood pressure, cannulation, venepuncture and catheterisation. There is therefore value in scheduling inductions before doctors report for their first day on the job.
Introduction
There is a serious gap in the clinical competence of newly qualified doctors, with up to 96% failing one or more clinical skills tests.1 Inductions before the first day on the job, which can be as short as just 1 day, could be the solution to what has been dubbed the ‘August nightmare’ in the UK. This is when new foundation house officers begin work, resulting in serious reductions in patient safety and patient care,2 as well as an increase in patient mortality rates,3 in the month of August. The 1-month risk period is created by the policy and tradition of newly qualified doctors receiving clinical inductions within 1 month of starting the job.4 Timing the clinical inductions as late as a month into the job can be avoided and it can be beneficial. There is promising evidence of a dramatic improvement in the clinical competence of new foundation house officers who completed an experimental induction before starting work.1,5–7 In one study, their failure rate reduced to 27% overall, and 0% on some skills, immediately after a 5-day induction.1 There is, therefore, a need to assess published evidence that might support the case for widening access to such inductions before doctors’ first day seeing patients.
Methods
A search was conducted for published reports of experimental clinical skills inductions. This is not a systematic review (see Preferred Reporting Items for Systematic Reviews and Meta-Analyses [PRISMA] guidelines)8 but some criteria guided the search process. To allow discovery of experiments reported in any subject field, the searches were run using Google Scholar. The search words included: ‘clinical skills inductions’; ‘clinical inductions’; ‘new medic induction’; ‘new doctors’ inductions’; ‘experiment inductions medic’; and ‘newly qualified doctors inductions’. The term ‘medic’ was inserted to allow ‘medical’ and ‘medicine’ to appear within the search results. The criteria used to select articles from the search results were as follows.
First, the data needed to come from a quantitative test of a clinical skills induction. Second, experimental subjects needed to be would-be foundation house officers, formerly called pre-registration house officers (PRHOs). Third, the experiment needed to have a within-subjects design, meaning that it involved identical pre-experiment and post-experiment tests of the new doctors’ competence level, or an acceptable between-subjects comparison. Fourth, the authors of the publication needed to have reported the new doctors’ mean competence level for each clinical procedure before and after the induction. Fifth, the authors needed to have used comparable methods of measuring the new doctors’ competence before and after the induction. Finally, the context of the experiments needed to be comparable in terms of educational standards or resources (for example, a study of new medics in a country suffering post-war instability and conflict was not included). The mean scores and p-values from the three articles1,5,6 that met all of the above criteria were then extracted.
Results
Tables 1–Table 3 show the average level of clinical competence among newly qualified doctors before the induction and immediately afterwards in the three experiments that met all the criteria.1,5,6 Two experiments1,5 also measured clinical competence 1 month after the induction, by which time the respondents had gained clinical experience as foundation house officers.1,5 Another experiment6 included comparison data from new doctors who were not inducted but who had accumulated 6 months’ clinical experience.
In one experiment,1 21 newly qualified doctors in the UK who were about to begin a PRHO job completed a 5-day induction. The title of ‘Foundation house officer’ has since replaced that of ‘PRHO’. The induction comprised breakfast-timed formal lectures covering the General Medical Council's induction curriculum; shadowing the outgoing PRHO; and objective structured clinical examination (OSCE) assessments carried out on three occasions over the course of the induction, with remedial intensive training if the OSCE was not passed. An additional measure of competence was the percentage of the new doctors who felt confident that they could complete each procedure before and after the induction.
After that 5-day induction,1 there were significant increases in the percentages of new doctors who felt confident in administering intravenous drugs (+14%), inserting an intravenous line (+40%), and taking blood (+8%). Each of these skills improved further after 1 month of clinical work as a foundation doctor (+27%, +17% and +4%, respectively). The percentage who felt confident about prescribing decreased significantly after the induction (–7%), and there was no significant change in confidence about certifying death (–4%), although it should be noted that these two skills were learnt during the inductions through shadowing with no remedial training or assessment. One month into the PRHO job, these percentages improved by +54% and +43%, respectively, implying that prescribing and death certification require a longer induction of 2 weeks5 or else 4 weeks of clinical work.
The effects of the 5-day induction on examined competence were similar to those on doctors’ confidence.1 The percentage of new doctors who failed at least one OSCE was 96% before the induction and 27% afterwards. Before the induction, the majority of the new doctors failed the blood pressure and cannulation OSCEs; afterwards, only six failed the blood pressure OSCE and none failed the cannulation OSCE. After the induction, there were no fails on catheterisation and venepuncture assessments. There were further improvements in all procedures after 1 month of clinical work as a PRHO, at which point no doctors failed any of the OSCEs.
In another experiment,5 50 would-be PRHOs attended a 2-week induction labelled a ‘Preparation for Practice Course’. The 2-week induction comprised a 1-day advanced life-support resuscitation course; a 1-day acute life-threatening events course; a 2-day life-threatening events recognition and treatment course; training in other aspects, such as infection control; a half-day shadowing the outgoing PRHO; helping the team; and assessment in a skills laboratory. The researchers5 measured the would-be PRHOs’ level of self-rated anxiety or confidence about completing a specific clinical procedure on day one of the induction, and then again on the last day of the induction through a 17-item questionnaire. Confidence with regard to each skill was scored out of 5 in a reverse-coded scale, in which the lower the score the higher is the level of confidence.
After this 2-week induction,5 there were significant increases in the level of self-rated confidence at taking blood, inserting an intravenous line, certifying death, prescribing and assessing patients and initiating treatment out of hours. These data contrasted with the lack of improvement in confidence in certifying death and prescribing reported after the 5-day induction.1 The significant effects of the 2-week induction suggest that a lengthier induction is necessary to improve confidence in these particular skills. After 1 month of working as a PRHO, there were further improvements in confidence relating to taking blood, intravenous line insertion, prescribing and out-of-hours work.
In the third experiment,6 10 first-year postgraduate house surgeons (PGY1s) in New Zealand, who had not yet started clinical work or received clinical skills training, completed a 1-day simulation workshop. Their competence at each skill was assessed through questionnaire responses self-scored from 0 (‘never seen, don't know how to do’) to 5 (‘competent to do alone’) at the start of the quarter, and again after the induction. The experiment also contained a between-subjects element; data were recorded from a comparison group of 10 PGY1s who had 6 months’ ward experience but no clinical skills training. The competence score for each skill was the sum of participants’ self-rated competence out of a maximum sum of 50.
After the 1-day induction,6 there were significant improvements in self-rated competence at lumbar puncture and spirometry. Compared with inducted doctors, doctors who had never been inducted but had 6 months’ clinical experience reported a lower level of competence with lumbar puncture and spirometry. There appeared to be a ‘ceiling effect’ with catheterisation, intravenous cannulation, arterial blood gas and electrocardiogram: the new doctors’ competence at these procedures was very high to begin with. Neither the induction nor 6 months of clinical experience raised competence much further. The induction had no significant effects on doctors’ confidence in their ability to perform ascitic tap or pleural tap, to measure central venous pressure or continuous positive airway pressure, or to use ventilators or chest drains; the scores remained low in all instances. Six months of clinical experience appeared to produce better confidence in inserting chest drains and in performing pleural tap and ascitic tap, but produced worse confidence in nasogastric tube insertion.
Table 4 includes data from an article7 that did not meet all the criteria because it combined data from medical and nursing students; discussions about the implications of these data for new medical doctors are tentative. There were 113 students inducted, of whom 41 were medical students. The experiment tested a 3-week induction involving clinical skills sessions and learning in mixed profession or uni-profession groups. Irrespective of the learning context, there were significant improvements in self-rated competence in catheterisation, venepuncture, use of electrocardiograms, medical calculations, basic life support, intravenous drug administration and blood pressure measurement as a result of the induction.
Discussion
Different experimental inductions produced significant improvements in new doctors’ level of clinical competence.1,5–7 There were improvements in cannulation, catheterisation, intravenous drug administration, intravenous line insertion, lumbar puncture, measuring blood pressure, the ability to assess patients and to initiate treatment out of hours, spirometry and venepuncture. Competence at prescribing and certifying death also improved after a 2-week induction. Effects on self-rated competence1,5–7 were corroborated by effects on examined competence.1 These experiments show a worthwhile approach to inducting would-be foundation house officers, with important practice and policy implications. They have pinpointed the clinical skills that can be improved by inductions and have provided support for the proposal2 to consider ways of preventing the risk to patient safety that arises when new foundation house officers start their jobs but have not yet been inducted. This discussion has shown that would-be foundation house officers can be successfully inducted before their first day on the job, with the effect of increasing their clinical competence. This is particularly pertinent in acute medicine.
The decline in average medical staff efficiency after new doctors start work is what poses the risk to patients. New doctors’ inexperience with tasks and the time spent by others providing guidance means that there are general efficiency losses in August, and this is what raises the risk of patient mortality. A systematic review of 19 studies found an average decrease in efficiency of between 0.3% and 7.2% after new doctors started work.9 Efficiency was assessed by measurements of the duration of clinical procedures and by operating room times, as well as using ‘spillover’ efficiency indicators such as the average length of patients’ stay. There is also an increase in the volume of urgent and rapid-response tasks logged by nurses in the month after new doctors start work, particularly out of hours;10 each task takes a median of 1 hour to complete and so the cumulative drain on staffing can increase the risk of patient mortality. Inductions could therefore provide an important solution to ameliorate the significant loss in average efficiency after new doctors start work.
It is important to acknowledge that the acceptable level of clinical competence in undergraduate exams is unavoidably different from the level of clinical competence expected professionally (in foundation year 1 and onwards) because of the extensive learning objectives of the undergraduate medical curriculum. Unless there is a change to the undergraduate curriculum, the clinical competence of an experienced or inducted foundation house officer cannot be expected to be achieved by undergraduate medical students. In the context of the current curriculum, the experimental inductions should be defined as workplace-learning events; they should provide a structured context in which new foundation house officers can gain accelerated clinical experience through shadowing, rehearsing skills and polishing their knowledge of procedures. This kind of learning, given the benefits for staff efficiency and patient safety, is an essential part of professional development. A recent major policy development (the Greenaway report11) pertinently emphasised the responsibilities of organisations in postgraduate medical education; the onus should therefore be on the NHS and other employers to provide inductions in a timely way.
Limitations
One challenge that arose in searching the evidence was the serious lack of data on foundation house officers’ clinical competence before and after their standard NHS inductions. Additional data would have been valuable, given the small sample sizes in the studies discussed and the differences in the lengths and content of the inductions for which data are available. A large random sample of foundation house officers’ pre-induction OSCE scores should be compared with their post-induction scores – with each evaluated under similar test conditions and taking into account the GMC curriculum and the level of competence expected at foundation house officer level. The experiment6 showing the benefits of an induction for lumbar puncture skills occurred in New Zealand, but this is not a clinical procedure frequently expected of new foundation house officers in the UK NHS. New OSCE data would inform on the success of existing induction arrangements, fulfil the need for more data about examined competence (especially given the small sample sizes in the experiment that included OSCEs), and infer ways in which NHS inductions can incorporate the format of the experimental inductions discussed here. Many NHS inductions involve an online format that 49.1%2 of doctors feel is ineffective or very ineffective.
Second, efficiency data will provide the missing link that explains when (and why) patient mortality rises after new doctors start work. More research into experimental inductions is needed, to allow a meta-analysis of the role of induction length, sample characteristics and efficiency gains. A systematic review found that four out of the six studies with samples large enough to detect a significant change in patient mortality from the baseline level of 2.7% found a significant increase in mortality of between 4.3% and 12.0% as a result of new doctors joining the staff.9 Measures of efficiency before and after an induction are therefore needed. Additionally, context can determine whether or not there is a decline in efficiency after new doctors start work. In some contexts, clear measures are already in place to maintain efficiency at an optimum level; for example, there is no evidence of a significant rise in ICU patient mortality after new doctors start work.12 Other evidence likewise suggests that surgery presents a smaller risk of the ‘August effect’ (the ‘July effect’ in the USA) than the general average of a 6% increase in mortality3 (or 4.3–12.0%9). An analysis of data from 2,920 patients who underwent metastatic spinal surgery in hospitals in the USA found a 3.3% increase in patient mortality in the month after new doctors started work.13 Additional published experiments are therefore needed to take into account both efficiency losses or gains and the context of the mortality. In determining the net benefits of inductions, the financial and staffing costs of inductions should be also considered. Authors of unpublished manuscripts or manuscripts in preparation are requested to contact the author for inclusion of their data in a meta-analysis.
The third limitation is that inductions offer a stopgap solution, whereas a more comprehensive solution would be to modify the undergraduate curriculum. Inductions have a comparatively small cost and require less change. All the same, workplace inductions cannot substitute for the need for changes in undergraduate medical education, especially if the proposal by Health Education England and the Medical Schools Council to make new graduates eligible for full GMC registration is implemented.14
Fourth, there is the question of whether competence before starting work is predictive of future clinical errors. Many studies demonstrating the predictive validity of OSCE scores present data about evaluations by others. One study found that PGY1s’ OSCE scores during their first month of work significantly predicted performance over the subsequent 18 months (as evaluated by academics).15 Furthermore, a meta-analysis of five studies found a significant correlation between prior OSCE scores and ratings of new doctors’ performance by their supervisors or by consultants.16 The moderate coefficients (0.3016 to 0.4915) found suggest that the initial ability of a new doctor is a significant but not perfect predictor of their future job performance. Data about clinical task errors during foundation year 1 would also offer a useful test of the predictive validity of OSCE scores and confidence ratings after the induction.
Conclusions
The discussion of competence in performing various clinical procedures before and immediately after different experimental inductions showed that there was an overall significant improvement in clinical competence as a result of the experimental induction. Experimental inductions have been shown to improve clinical competence significantly, and thus there is scope to translate them into standard inductions, the completion of which is mandatory for all would-be foundation house officers before their first day on the job. Certainly, there are resourcing and policy challenges, and the long-term solution could be to modify the undergraduate medical curriculum. A stopgap solution is to offer inductions to all would-be foundation house officers whose OSCE scores in key procedures (within their final degree results) were below a specific level. Overcoming the roadblock of policy will need a discussion of the gains and costs of rescheduling inductions.
- © 2014 Royal College of Physicians
References
- ↵
- ↵
- Vaughan L,
- McAlister G,
- Bell D
- ↵
- Jen MH,
- Bottle A,
- Majeed A,
- Bell D,
- Aylin P
- ↵
- NHS employers
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Blakey JD,
- Fearn A,
- Shaw DE
- ↵
- Greenaway D.
- ↵
- ↵
- ↵General Medical Council Education and Training Advisory Board. The timing of full registration, 2013. www.gmc-uk.org/3___The_timing_of_full_registration.pdf_54171569.pdf [Accessed 13 March 2014].
- ↵
- ↵
Article Tools
Citation Manager Formats
Jump to section
Related Articles
- No related articles found.
Cited By...
- No citing articles found.