Learning to make a difference: introducing quality improvement methods to core medical trainees =============================================================================================== * Emma Vaux * Susan Went * Martin Norris * Jane Ingham ## Abstract The ‘Learning To Make a Difference’ (LTMD) initiative was a Royal College of Physicians/Joint Royal Colleges of Physicians Training Board collaboration supported by The Health Foundation. It aimed to support the learning and development of new and relevant skills in quality improvement (QI) methodology by trainees to enable them to deliver effective QI projects at the frontline. Core medical trainees in five deaneries were offered the opportunity to undertake a QI project in place of a mandatory clinical audit during 2010–2011. In total, 61 trainees completed 46 QI projects. Evaluation of the project outcomes demonstrated the acceptability, feasibility and strengths of trainee-led small-scale change and how this can deliver improvement in the quality of multidisciplinary working, clinical practice and patient care. The LTMD project supports the further development and spread of this approach, encouraging all physician trainees, and their supervisors, to understand, develop and embed appropriate skills in QI methodology as part of their professional role. In addition, the project has identified the necessary infrastructure to enable this to happen. Key Words * quality improvement * service improvement * trainee * life-long learning ## Introduction Continuous improvement in the quality of patient care is a driving principle for teams of health professionals and support staff working together at the frontline. Delivering high-quality patient care requires a range of complementary strategies and multifaceted and multilayered interventions. A wide variety of improvement strategies are required. These range from the implementation of recognised national guidelines, clinical audit, peer and practitioner review, validation and service accreditation to quality improvement (QI) models drawn from industry. Through the evaluation of trainee physicians, the Joint Royal Colleges of Physicians Training Board (JRCPTB) and the Royal College of Physicians (RCP) became aware that the requirement ‘to do an audit’ had, in many cases for junior doctors, been little more than a data collection exercise with no subsequent learning, action or long-term consequences for patient care. The ‘Learning to Make a Difference’ (LTMD) project has specifically explored the acceptability and feasibility of introducing a QI project within core medical training, and tested one method for engaging and inspiring frontline junior doctors. The aim of the initiative was to enable trainees to deliver change in practice through the acquisition of new knowledge and the practical application of skills within a QI project using the model for improvement.1 The definition of QI used in this work was ‘better patient experience and outcomes achieved through changing provider behaviour and organisation through using a systematic change method and strategies’.2 There is good evidence that better quality, safer care is more efficient and improves patient outcomes and experience while delivering better value for money.3,4 The aim was to enhance the trainee's own learning and skills while bringing value to their organisation and make a visible difference to patient care within a short timeframe. ## Method The model for improvement was fundamental to the LTMD initiative; it was both the framework for the pilot itself and the methodology used by the trainees for their quality improvement projects. ### Model for improvement The model for improvement using the three core questions (What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in an improvement?) and the ‘Plan Do Study Act’ (PDSA) cycle provided a simple structured framework for the LTMD project design, development and delivery (Fig 1).1 ![Fig 1.](http://www.rcpjournals.org/https://www.rcpjournals.org/content/clinmedicine/12/6/520/F1.medium.gif) [Fig 1.](http://www.rcpjournals.org/content/12/6/520/F1) Fig 1. The Model for Improvement. Associates in Process Improvement (1996)1 The simple structured framework for the �Learning to Make a Difference�, project design, development and delivery. *What are we trying to accomplish?* To start trainees on a pathway for lifelong evaluation and QI of the service that they deliver, by developing and embedding new skills in QI techniques. To identify the framework and infrastructure required for the successful implementation of this strategy. *How will we know that a change is an improvement?* By evaluating the value, acceptability and feasibility of this change to the trainee (through assessment of their learning and development change to their practice, their team work or their patient care), their organisation and, most importantly, to the patient. *What changes can we make that will result in an improvement?* To offer core medical trainees (CMT) the opportunity to undertake and complete a QI project in place of an audit in a 4–6 month training period. ### Identifying study participants The LTMD project started during August 2010 and finished during May 2011. Five deaneries participated in the project: Kent, Surrey and Sussex; Northern; North Western; Oxford; Yorkshire and Humber and South-Eastern Scotland. The deanery leads were CMT training programme directors identified through the RCP CMT Advisory Committee. The leads took forward recruitment to the project within their deanery. Methods of recruitment and approaches to trainees and their consultant supervisors were left to local determination. Different approaches were used and included: direct contact with consultant supervisors and/or CMT trainees; regional presentation of the planned project; individual and face-to-face recruitment, generic and/or personalised emails and use of other media, such as telephoning and/or texting. ### Implementation of QI project Participating trainees were asked to identify a topic for team or individual improvement. Each trainee was allocated a consultant supervisor. They were supported to identify the aims, measures and changes to be tested and required to complete and report on at least one small-scale test cycle within their 4–6 month post. ### Resources to support the trainee and supervisor The participants did not attend any specific training course. The resources developed included a PowerPoint presentation, the LTMD practical toolkit, monthly newsletters, LTMD website and the resource of the local deanery lead supported by the central implementation team.5 The LTMD toolkit adopted and adapted excellent work done by others, including the Building Safety Improvement Skills (BaSIS) programme of the NHS,6 Institute for Healthcare Improvement (IHI) tools,7 *British Medical Journal (BMJ)* audit repository8 and The Kings Fund.9 A motivational film to promote and sustain engagement in the project was produced and directed by seven of the CMT trainees from the Oxford Deanery and made available to all involved.5 QI mentors (medical consultants with expertise in QI) were recruited for each deanery and put in touch with the deanery leads. The aim was that deanery leads would partner with the QI mentors to help ensure the trainees’ projects were supported to use QI methodology in the most effective way. The LTMD project was supported by a small central implementation team, including a clinical lead, project manager and an overarching steering group (members from the RCP, JRCPTB, The Health Foundation, NHS Institute for Innovation and Improvement, Department of Health, *BMJ* and a junior doctor). ### Presentation of completed QI projects Each deanery organised a regional event for their trainees to present their work in front of a panel of local judges in April 2011. The best project(s) from each deanery were presented in oral or poster format at a national event in May 2011 before the steering group. The winning QI project has been supported to present at the 2012 International Forum on Quality and Safety in Healthcare ### Project evaluation Reporting and assessment mechanisms were developed to enable evaluation of the process, outcome and balancing measures, and were augmented by semi-structured interviews and qualitative feedback from all participants. Kirkpatrick's model for full and meaningful evaluation of learning and training was applied retrospectively to provide a framework to report the results (Table 1).10 The model has been adapted for evaluation contextualized to healthcare and has been used in a recent systematic review of the impact of workplace-based assessment on doctors’ education and performance.11,12 View this table: [Table 1.](http://www.rcpjournals.org/content/12/6/520/T1) Table 1. Methods of evaluation of the LTMD project using Barr's adaptation of Kirkpatrick's model for evaluation.11 The DICE score is a scoring measure that aims to provide an objective view of whether a project is likely to succeed, and is aimed at leaders within the environment of the project. Leaders within the project were asked to score the overall LTMD project against each of the four factors that comprise the DICE framework: Duration of the project; the performance Integrity of the team; the organisational Commitment to change; and the additional Effort required of staff members. Results place a project within one of three self-explanatory zones: the win zone, the worry zone and the woe zone.13 LTMD project progression was assessed using a five-point Likert scale, with descriptors derived from the IHI improvement project assessment model (1: forming a team to 5: achieving outstanding sustainable results).14 Qualitative assessment was made through questionnaire and semi-structured interviews with the participants. ## Results In total, 61 trainees completed 46 QI projects (Table 2). Collated trainee QI project themes are given in Box 1. #### Box 1. QI project themes. * Prescribing eg anticoagulation, oxygen, in Parkinson's disease, in diabetic ketoacidosis * Prevention and/or improved management of pneumonia * Medical handover * Sharps boxes and safety awareness * Communication eg multidisciplinary team, hospital at night, general practitioner referrals * Discharge planning * Routine methicillin-resistant Staphylococcus aureus swabs in the clinical decisions unit * Reducing unnecessary investigations and/or procedures * Implementation of best practice View this table: [Table 2.](http://www.rcpjournals.org/content/12/6/520/T2) Table 2. Number of projects and CMT trainees involved by Deanery. The adapted Kirkpatrick Model for assessing the response and impact of this pilot was applied retrospectively. The model for improvement was the framework used for the LTMD project. The process, outcome and balancing measures are given in brackets. ### Level 1: particpants’ reactions These outcomes relate to participants’ views of their learning experience and satisfaction with implementing this approach. DICE Score13 (process measure): pre-project: 83% of respondents placed the project in the win zone and 17% in the worry zone (n=6). Post-project: 100% of respondents placed the project within the win zone (n=5). The respondent who had put the project in the worry zone at the start placed the project in the win zone on completion. ‘Getting Started’ evaluation of recruitment of trainees and supervisors to LTMD project (process measure): different methods of recruitment to the LTMD were used by the local deanery leads. Overall, 10 (of 58) consultant supervisors and eight (of 70) trainees when asked did not agree to participate (Table 3). The reasons given by the supervisors included ‘not sure what it's all about and too much else on’, ‘lack of time’ or ‘no response from a Trust email’, and from trainees ‘felt that they had priorities with getting the MRCP’, ‘already doing an audit’ and ‘no response from a Trust email’. In total, 16 trainees did not complete a project having agreed to participate. The reasons given included ‘difficulty getting kit together for project, loss of interest, exam priorities’, ‘apathy, lack of time’ and ‘unable to think of worthwhile project.’ One trainee failed to complete a project. View this table: [Table 3.](http://www.rcpjournals.org/content/12/6/520/T3) Table 3. ‘Getting Started’ evaluation of recruitment to the LTMD project. LTMD project progression assessment scale (process measure): on project conclusion, self-assessment by the central LTMD team positively evaluated the progress made by the pilot in achieving the intended improvement change for the aim, outcome and process measures (Table 4). View this table: [Table 4.](http://www.rcpjournals.org/content/12/6/520/T4) Table 4. Project Assessment Scale of LTMD project: self-evaluation of project progress. Evaluation of LTMD local and central support (process measure): qualitative assessment through questionnaire (69 trainees and supervisors responded (of a total of 97)) and semi-structured interview of the five deanery leads highlighted areas that were crucial to success. Enthusiasm, commitment and engagement with this approach were not enough to achieve success unless supported by knowledge and understanding of QI methodology. A face-to-face, personalised approach at trust level to engage and support the trainees from start to completion was effective, as was providing examples of QI projects to make sense of the process. Trainee-led ideas as the stimulus for QI projects, with multidisciplinary involvement, were the most successful. Alignment with a trust QI agenda further enhanced the value to all the participants. Getting the right local supporting infrastructure for the trainees and supervisors was identified as being crucial in enabling the delivery of effective projects. Professional leadership and central support was crucial in supporting the pilot sites and ensuring sustainability of this approach. ### Level 2: acquisition of knowledge, skills and attitudes The outcomes here relate to what knowledge and skills were learned. The QI project assessment tool (QIPAT) (outcome measure): this tool was designed to assess trainee competence in developing new skills and knowledge and completing an LTMD QI project. It was completed by the supervisor with the trainee on completion of their project and could be uploaded to their e-portfolio. The content of the completed QIPAT tools was not assessed as part of this project. ### Level 3: changes in behaviour This level covers behavioural change transferred from the learning environment to the workplace prompted by modifications in attitudes or perceptions, or the application of newly acquired knowledge and/or skills in practice. Evaluation of acceptability and feasibility of doing QI projects among all participants (outcome measure). This was measured on a four-point Likert scale with two questions for consultant supervisors and four questions for trainees. Supervisors strongly endorsed the project and expressed enthusiasm for continued involvement in QI work (Table 5). The trainees expressed enthusiasm for completing further QI projects: 91% (31/34) would undertake another QI project and 100% found the project a valuable exercise (Table 6). Given that this assessment was undertaken once, it was not possible determine whether this behaviour was sustained over time. View this table: [Table 5.](http://www.rcpjournals.org/content/12/6/520/T5) Table 5. Evaluation of acceptability and feasibility of the consultant supervisor participation in the LTMD project. View this table: [Table 6.](http://www.rcpjournals.org/content/12/6/520/T6) Table 6. Evaluation of acceptability and feasibility of CMT trainee participation in the LTMD project. ### Level 4: changes in organisational practice and benefits to patients This relates to wider changes in the organisation and/or delivery of care, attributable to an education programme. Local impact on patient care (outcome measure): 34 trainees representing 41 project teams responded. Of the 34 trainees responding, 88% (30/34) assessed their QI project objectives as achieved. In total, 14 of the 36 supervisors responded. Of the 14 supervisors, 93% (13/14) assessed the QI project objectives as achieved. In addition, 85% (29/34) of trainees and 88% (12/14) of supervisors assessed their project as having had a significant impact on improving clinical practice. Realising that not every change would result in improvement was important learning for the trainees; instead, the emphasis was how to apply repeated small cycles of change to deliver the goal. Changes in patient care and return on investment (outcome measure): the emphasis in the LTMD project was on trainee learning and development of new skills in QI methodology. However, demonstrable impact on patient care and return on investment was demonstrated by individual QI projects. An example of one such project was on the prevention of hospital-acquired pneumonia (HAP), with an estimated financial cost saving of 45 bed days and £1,800 antibiotic costs per week and, more importantly, a human cost saving of nine patients per week being prevented from developing HAP.15 There was no evidence that the time spent on a QI project had a negative impact on the other commitments of trainees (balancing measure). However, the process for facilitating and delivering a mandatory clinical audit was not in place in some trusts and, as a result, for some trainees ‘doing a QI project’ was perceived as an additional activity to their ‘usual’ training requirements. ## Discussion The LTMD project was a test of the feasibility and acceptability of implementing QI methodology into practice as part of core medical training and of the resources needed to deliver that methodology. The test was applied to a small sample, in real time, across a variety of clinical settings and across a short timescale to make a pragmatic evaluation of outcomes. There are many examples of best practice that do not translate into everyday use, with widespread failures in adherence to clinical management guidelines, variations in practice and resource use. ‘Learning by doing’, testing and experimentation are valid ways of increasing knowledge and they are all techniques widely used for acquiring the knowledge to negotiate daily life successfully. The aspiration that CMT trainees would develop and embed new skills and, at the same time, learn some simple and practical QI techniques to take forward in their clinical practice seems to have been borne out. In their evaluation documents, CMT trainees highlighted the valuable and meaningful role that a junior doctor can have in QI. QI was seen as a real-time, dynamic process by both the trainees and their supervisors involved in the LTMD project. Making focussed small-scale change through a simple structured framework enabled visible effective change to happen within the timeframe of a training post. Therefore, the value of involving frontline trainees in making a real difference to patient care cannot be underestimated. The rate of recruitment to QI projects was slower and occurred over a longer period than anticipated. Some of the factors contributing to this were: the importance of face-to-face contacts in the recruitment process; a continuing level of scepticism about QI when compared directly with research; multiple demands on supervisors’ and trainees’ time; and a lack of exposure to good-quality QI training. One of the core assumptions made at the beginning of the project was that trainees would be offered the option of undertaking a QI project instead of the required clinical audit. It became apparent that, in many cases, CMT trainees were not completing a clinical audit, only data collection. This has implications for any change in training requirements. The box-ticking attitude to clinical audit was apparent in all the deaneries and there is a risk that any alternative might suffer the same fate. Alignment of QI projects or audits to the Quality, Innovation, Productivity and Prevention programme of a trust would increase the level of organisational engagement and be influential. The value of adopting QI methodology in enhancing the ‘implementing change’ part of the clinical audit cycle and enabling trainees to recognise how dynamic junior doctor-led audit could be, has been missed to date and this is a potential opportunity to enhance the current trainee clinical audit process. The most successful project outcomes originated from trainee-led ideas that were related to the strategic goals of a clinical team or trust with multidisciplinary involvement, where the trainee was supported by an engaged consultant supervisor. The trainee knew that their project mattered, not only in terms of their own development, but also to their local deanery lead and to their trust. Central support from the RCP/JRCPTB working group reinforced this essential motivator to their achievements. Where a QI mentor became involved in facilitating a QI project, an estimate of their time was approximately 4 h over the whole 4–6 month project. Developing a network of trust QI consultant champions, each acting as an advocate for the junior doctors at their trust, and providing the necessary support and knowledge resource, is seen as being core to the effective expansion and sustainability of embedding QI in junior doctor training, as well as being practicable in delivery. In addition, the tools and resources designed and developed as part of the project are now available to all through the LTMD website. The LTMD approach stands up well in comparison to other evaluations of workplace-based assessments that are currently in use in the UK (eg Mini-Clinical Evaluation Exercise (Mini-CEX) and Direct Observation of Procedure or Skills (DOPS)).12 With changes in patient care evident from some of the trainee QI project outcomes, LTMD has demonstrated more evidence of benefit to patients compardcompared with any of the current workplace-based assessments. ### The next steps The LTMD project evaluation demonstrates acceptability and feasibility among participants; CMT trainees and consultant supervisors consider QI as having practical application to their delivery of clinical practice. QI methodology has a broad application to making change in practice, whether the change is driven by results of data collection, benchmarking against peers or the need to improve local processes, local systems, patient experience and outcomes. The project supports the further development and phased spread of this approach, but the necessary curricular changes will take time. In the immediate term, deaneries are encouraged to offer trainees the choice of completing a QI project or a clinical audit within a training year. For trainees completing a clinical audit, a QI approach should be used in the ‘implementing change’ part of the clinical audit cycle. Work to develop a network of local trust QI champions, underpinned by development of a coherent framework and infrastructure to support all involved, is underway. In addition, the RCP Education Department has introduced a ‘Quality Improvement for Consultants’ education programme. ## Funding An unrestricted grant was provided by the RCP and The Health Foundation ## Acknowledgements We would like to thank Dr Mike Cheshire, former clinical vice-president RCP, Professor Bill Burr, medical director JRCPTB, Ms Winnie Wade, director of education RCP, and Ms Jane Ingham, director of clinical standards RCP for the original idea. We would like to thank all trainee and supervisor participants, local Deanery leads (Drs John Anderton; Karen Goodman, Nik Patel, Randy Smith and Emma Vaux), QI mentors (Drs Tom Downes, Daniel Higham, Mark Lambert, Carol Peden, Kevin Stewart, Paul Sullivan and Simon Watson) and Steering Group members for their assistance with this project. We would also like to thank RCP and Health Foundation for funding the project. * © 2012 Royal College of Physicians ## References 1. 1. Nolan K 1. Nolan T, 2. et al Langley G Nolan K Nolan T, et al. The improvement guide: a practical approach to enhancing organizational performance. 2nd edn. San Francisco: Jossey-Bass Publishers 2009. 2. Øvretveit J. Does improving quality save money? A review of the evidence of which improvements to quality reduce costs to health service providers. London: The Health Foundation 2009. 3. 1. MacIntosh-Murray A 1. Portcellato C, 2. et al Baker GR MacIntosh-Murray A Portcellato C, et al. High performing health care systems: delivering quality by design. Toronto: Longwoods Publishing 2008. 4. 1. Mendel P 1. Robert G Bate P Mendel P Robert G. Organising for quality: the improvement journeys of leading hospitals in Europe and the United States. Oxford: Radcliffe Publishing 2008. 5. RCP 2012. Learning to make a difference. [www.rcplondon.ac.uk/resources/clinical-resources/learning-to-make-a-difference](http://www.rcplondon.ac.uk/resources/clinical-resources/learning-to-make-a-difference) [Accessed 4 October 2012]. 6. NHS. BaSIS: building safety improvement skills, 2012. [www.institute.nhs.uk/safer\_care/safer\_care/basis%3A\_building\_safety\_improvement\_skills.html](http://www.institute.nhs.uk/safer\_care/safer\_care/basis%3A\_building_safety_improvement_skills.html) [Accessed 4 October 2012]. 7. Institute for Healthcare Improvement 2011. Plan-Do-Study-Act (PDSA) worksheet. [www.ihi.org/knowledge/Pages/Tools/PlanDoStudyActWorksheet.aspx](http://www.ihi.org/knowledge/Pages/Tools/PlanDoStudyActWorksheet.aspx) [Accessed 4 October 2012]. 8. British Medical Journal 2012. [www.bmj.com](http://www.bmj.com) [Accessed 4 October 2011]. 9. The Kings Fund 2012. Our work: commissioned leadership programmes. [www.kingsfund.org.uk/leadership/commissioned-work](http://www.kingsfund.org.uk/leadership/commissioned-work) [Accessed 4 October 2012]. 10. Kirkpatrick DL. Revisiting great ideas. Train Dev 1996; 50:54–9. 11. 1. Freeth D 1. Hammick M, 2. et al Barr H Freeth D Hammick M, et al. Evaluations of interprofessional education. A United Kingdom review for health and social care 2000. London: United Kingdom Centre for the Advancement of Interprofessional Education in Primary Health and Community Care; Southwell: British Educational Research Association 2000. 12. 1. Archer J Miller A Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. Br Med J 2010; 341:c5064.doi:10.1136/bmj.c5064 [Abstract/FREE Full Text](http://www.rcpjournals.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDEvc2VwMjRfMS9jNTA2NCI7czo0OiJhdG9tIjtzOjI3OiIvY2xpbm1lZGljaW5lLzEyLzYvNTIwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 13. 1. Keenan P 1. Jackson A Sirkin HL Keenan P Jackson A. The hard side of change management. Harvard Business Rev 2005. 14. Institute for Healthcare Improvement Assessment Scale for Collaboratives 2011. [www.ihi.org/knowledge/Pages/Tools/AssessmentScaleforCollaboratives.aspx](http://www.ihi.org/knowledge/Pages/Tools/AssessmentScaleforCollaboratives.aspx) [Accessed 30 October 2012]. 15. 1. Wain H 1. McDonald A, 2. et al Vaux E Wain H McDonald A, et al. Healthcare associated pneumonia is not inevitable. In: IHI 23rd Annual National Forum on Quality Improvement in Healthcare. Institute for Healthcare Improvement 2011. ## Uncited reference Institute for Healthcare Improvement, 2011. *Assessment scale for collaboratives*. [www.ihi.org/knowledge/Pages/Tools/AssessmentScaleforCollaboratives.aspx](http://www.ihi.org/knowledge/Pages/Tools/AssessmentScaleforCollaboratives.aspx) [Accessed 4 October 2012].