Development of a GMC aligned curriculum for internal medicine including a qualitative study of the acceptability of ‘capabilities in practice’ as a curriculum model
ABSTRACT
The Shape of Training review and the General Medical Council (GMC) requirements in demonstrating generic professional capabilities were major drivers for the Joint Royal Colleges of Physicians Training Board (JRCPTB) to develop the new internal medicine curriculum. In particular, the GMC required progression to a more outcomes-based curriculum.
The present curricula for physician training are based on demonstrating a large number of individual competencies that are assessed by a variety of different methods. It was felt that current system was overwhelming and had become a ‘tick box’ exercise. The new curriculum is based on a much smaller number of outcomes, called capabilities in practice, which reflect the key professional work activities of a fully trained physician. The aim is to re-emphasise the role of professional judgement in ‘trusting’ the work a trainee does and thus make assessment more realistic and meaningful for both trainees and trainers.
The proof of concept study explored the feasibility of using this outcomes-based model of assessment in a UK NHS setting. The learning from the study has enabled us to make significant changes to the internal medicine curriculum. The GMC has recently approved the curriculum and the JRCPTB is implementing the programme from August 2019.
- Competency-based medical education (CBME)
- entrustable professional activities
- Royal College of Physicians
- JRCPTB
- internal medicine curriculum
Summary
The Shape of Training review and the General Medical Council (GMC) requirements in demonstrating generic professional capabilities (GPCs) were major drivers for the Joint Royal Colleges of Physicians Training Board (JRCPTB) to develop the new internal medicine curriculum. In particular, the GMC required progression to an outcomes-based curriculum using capabilities in practice.
The proof of concept (PoC) study explored the feasibility of using this outcomes-based model of assessment in a UK NHS setting. The learning from the study has enabled us to introduce a novel assessment method into the internal medicine curriculum. The GMC has approved the curriculum and the JRCPTB is implementing the internal medicine curriculum from August 2019.
Introduction
The training of physicians in the UK needed to evolve in response to two major policy initiatives. Firstly, the Shape of Training review (2013) highlighted that the needs of patients are changing.1 Life expectancy is increasing and patients are diagnosed with multiple complex medical conditions, which are largely chronic diseases. The training of all physicians needs to match this changing clinical environment. Secondly, the GMC required all medical colleges to produce new curricula based around integrating their GPCs with an outcomes-based approach.2,3 In response the JRCPTB has developed a new UK internal medicine curriculum based on outcomes for physician training in order to ensure physicians are better equipped to cope with the complex needs of patients. Currently competency-based medical education (CBME) is used to train physicians in all specialties. However, CBME has remained controversial and the methods by which trainees are currently being assessed have often been criticised as a ‘tick-box approach’.4–7
In producing the new internal medicine curriculum, we have transitioned CBME to a simplified outcomes-based approach with the objective of improving assessment and making it more realistic and valuable for both trainees and trainers. This has been achieved by incorporating the concept of entrustable professional activities (EPAs),8 which we have called capabilities in practice (CiPs) in postgraduate physician training. This paper gives an overview of how we incorporated CiPs/EPAs into the new UK internal medicine curriculum. It includes our experiences of undergoing a pilot study with trainers and trainees testing the use of CiPs, the learning from the pilot and how we subsequently finalised the curriculum in order to meet the standards of postgraduate curricula as defined by the GMC.3 The UK stage 1 internal medicine curriculum was approved by the GMC in December 2017 and has replaced the core medical training (CMT) programme from August 2019.
Competency-based medical education, entrustable professional activities and trust
CBME was introduced in order to bring structure to training and assessment to ensure consistent quality in graduate outcomes, which was provoked, by patient safety concerns and demands for greater accountability.9 Advocates for CBME propose that it promotes relevant skills for medical training and better curriculum governance and by embracing competencies in a proper way will lead to positive healthcare.10
Despite these positive aspects of CBME, some have raised concerns and have identified numerous challenges when it comes to implementing CBME. The limitations of CBME have been described by Leung:
If applied inappropriately, it can result in demotivation, a focus on minimal acceptable standards, increased administrative burden and a reduction in the educational content.11
CBME has been described as checking boxes on checklists rather than assessing the outcome of training in preparation for practice, leading to a fear that educators may spend more time on administration of programmes rather than ensuring quality of learning experiences.12–14
The present curricula for physician training are based on achieving a large number of individual competencies that are assessed throughout training by a variety of different methods. In the UK, the core medical training curriculum can amount to 120 individual competencies requiring completion and sign off in order to finish the programme.7 A prevailing view is that the current system is overwhelming and the signing off of competencies has become a tick-box exercise. 4–7
There appears to be a ‘conceptual gap’ between signing off the list of competencies and the actual daily clinical tasks of physicians.14,15 In real life, competencies are not assessed individually, for example, a trainee who is assessed on ‘breaking bad news’ needs to demonstrate multiple competencies such as clinical experience, medication options, communications skills and professionalism. In this way, an outcomes-based assessment using a system of EPAs can be used to help bridge the ‘conceptual gap’ and reflect actual practice. EPAs incorporate multiple different competencies and replicate what happens in practice by linking the educational and medical worlds. EPAs have been described in the literature as ‘units of professional practice, defined as tasks or responsibilities to be entrusted to the unsupervised execution by a trainee once he or she has attained sufficient specific competence’.8 In order to bridge the ‘conceptual gap’ between educational theory and clinical practice, the concept of EPAs/CiPs can be incorporated in order to support and operationalise CBME.16,17
In a clinical teaching setting, ‘trust’ is key to help a supervisor decide when and for what tasks they can trust trainees to assume clinical responsibility. Clinicians routinely make risk versus benefit decisions when it comes to managing patients. This risk versus benefit decision also takes place when it comes to supervising junior trainees. Inserting a central venous catheter is much more risky than inserting a simple cannula in to a peripheral vein. Clinical supervisors initially observe new trainees and gauge quickly what tasks they can be ‘trusted’ to do. In day-to-day clinical practice, supervisors are frequently making these decisions when supervising junior trainees. For example in the case of managing an acutely unwell patient in the emergency department, a supervisor may only ask a junior trainee who they felt could be ‘trusted’ to manage this patient appropriately. In this case, of the acutely unwell patient in the emergency room, the level of supervision will vary from trainee to trainee. The senior physician may ‘trust’ an experienced junior trainee to manage this acutely unwell patient independently without any supervision. In contrast, if this was an inexperienced trainee, the senior physician may only ‘trust’ this trainee to manage the same patient with immediate direct supervision. In this way, a supervisor decides what task a trainee can be ‘trusted’ to do as well as determining the level of supervision required.
The example above is work-task specific so one trainee may be trusted to work unsupervised on managing an unwell patient however they may not be trusted to insert a central venous line without direct supervision. For each work task, a trainee can be assessed as having a specific trusted ‘level’ of supervision, which is decided by a supervisor. This approach to assessment builds on the normal day-to-day decisions that supervisors already make by defining progression in terms of the work that a trainee is trusted to do. Near the completion of training, doctors are formally entrusted at the Annual Review of Competence Progression meeting to undertake all the work tasks independently and act without supervision in all the areas required. Thus, EPAs/CiPs are arguably more likely to mirror the sorts of decisions clinical teachers make about their learners all the time in clinical practice.9 In the literature, there are four sets of variables that determine when entrustment decisions are made, or, in other words, when risks seem acceptable. These are perceived trustworthiness, perceived risks, perceived benefits and trust propensity. These trainee and supervisor dependent factors are important in making any entrustment decisions.18 This concept of trust and entrustment is important to understand when using EPAs/CiPs as a means of assessing trainees.
Once all the essential professional activities that a trainee must achieve in a curriculum have been described, multiple EPAs/CiPs can be developed. These can then be used in assessing physician trainees as they progress through training. Once they have mastered all the EPAs/CiPs they can then be ‘signed off’ by the training programme director and are ready to apply for a certificate of completion of training in their specialty.
The internal medicine curriculum and the proof of concept study
Throughout the international medical education community there has been a shift towards adopting EPAs/CiPs within undergraduate and postgraduate curricula, however there are few examples of curricula that have actually implemented EPAs/CiPs formally into physician training. Based on the literature and a consultative process with the main UK physician educators (heads of the postgraduate schools of medicine and specialty advisory committee chairs), 14 CiPs for internal medicine were initially described that collectively covered the key professional activities expected of a fully trained physician (Table 1).19
Within each CiP there are four ‘levels’ at which a trainee might be judged to be performing. Ascribing a specific level to each CIP should allow progress to be assessed and to relate progress to the tasks which are actually performed in the clinical workplace and the level of supervision under which that task must be performed (Tables 2 and 3).20
The PoC study explored the feasibility and acceptability to trainers and trainees of using an outcomes-based model of assessment in a UK NHS setting.19 The study assessed whether supervisors could make entrustment decisions using CiPs in a range of specialties and how this outcomes-based assessment system compared to the system currently being used. We also looked at what kind of evidence (eg workplace-based assessments (WPBAs), NHS's ePortfolio evidence and courses) would be required in order for supervisors to be able to ascribe a level to each CiP. The levels of supervision described above were also tested to assess whether they were meaningful, useful and helpful to trainers and trainees for all the proposed 14 CiPs.
Methods
Participants in the study included trainees (core medical trainees and higher specialist trainees), clinical supervisors and educational supervisors. Invitations were sent to all heads of school and training programme directors via an alert on ePortfolio. The specialties that were invited are listed in supplementary material S1. Participants volunteered and completed a study form and were confirmed for the study by the JRCPTB.
There was online training through guides and podcasts and some trainers and trainees had face-to-face training in how to practically use this form of assessment. The face-to-face training sessions were attended by 33 participants (supervisors and trainees). The following tools were produced (see supplementary material S2 and S3).
A CiP study trainee self-assessment form (supplementary material S2).
A CiP study educational supervisor form (supplementary material S2).
A CiP study clinical supervisor form (supplementary material S2).
An evaluation form for each participant by which the views of participants could be quantitatively and qualitatively assessed (supplementary material S3).
The CiP study forms described above were available on ePortfolio and involved participants scoring a level of supervision for each of the 14 CiPs. Participating trainees were linked with their clinical and educational supervisors in their ePortfolio and were asked to engage in the process summarised in Fig 1.
Trainees were asked to:
consider what evidence they needed to provide to inform decisions about their performance at each CiP
complete a CiP study trainee self-assessment report. This involved the trainee individually scoring a level of supervision, at which they felt they were performing for each CiP.
Clinical supervisors were asked to:
review the trainee's ePortfolio and consider their personal experience of the trainee
complete a CiP study clinical supervisor report indicating what level they felt the trainee was performing at for each CiP
meet with the trainee to complete the report.
Educational supervisors were asked to:
review the trainee's ePortfolio and consider the clinical supervisor report and trainee self-assessment
meet with the trainee in order to discuss progress and document a level of performance of each CiP on the CiP study educational supervisor form.
All participants in the PoC study were asked to complete a feedback questionnaire through SurveyMonkey. The evaluation questions asked about the process, the levels of supervision, how this compared to the current method of assessment and whether there was a need for training. Both quantitative and qualitative questions were asked. A thematic analysis process was used to code themes identified in the feedback questionnaire.
The findings from the PoC study were widely presented to a variety of stakeholders, the stakeholders are summarised in supplementary material S4. The JRCPTB and education department held a separate development day with these participants, which included a workshop to assess and solve issues that had arisen from the findings of the PoC study. Participants were asked to take the role of educational supervisors to complete CiP forms using hypothetical trainee WPBA forms and hypothetical clinical supervisor reports. Participants were asked to complete an evaluation form, which was analysed and used to develop the outcomes-based model of assessment for the internal medicine curriculum.
Results
Proof of concept study
In total, 78 participants (30 trainees, 28 educational supervisors and 20 clinical supervisors) engaged in the process and completed the feedback questionnaire.
Participants were asked the following questions and the results are displayed below each question that was asked (Tables 4–8).
Table 9 summarises the most common themes that were encountered following a qualitative thematic analysis.
From the summary of the common themes arising from the evaluation there were many positive themes found in the study. Free text comments from trainees and supervisors are described in Table 10.
However, there were also some aspects where participants have found problems. These themes are discussed in further detail in Table 11.
Development day focus group
The PoC study highlighted that there were issues in using and understanding the initial levels of supervision (Tables 2 and 3). These levels of supervision were reviewed and the literature further examined. The Association of American Medical Colleges in their Curriculum developers’ guide have described entrustment levels divided into direct and indirect supervision.21 The refined levels of supervision in Table 12 have been adapted from the Accreditation Council for Graduate Medical Education definition of supervision.22
In order to pilot the use of these new level descriptors, JRCPTB and the education department held a focus group with 37 participants comprising SAC chairs from various physicianly specialties, heads of school and trainees. Participants were asked to take the role of educational supervisors to complete CiP forms using hypothetical trainee WPBA forms and hypothetical clinical supervisor reports. The major themes arising from the development day are summarised in Table 13.
Discussion
The qualitative analysis from the PoC study provided evidence from trainers and trainees that an outcomes-based assessment using CiPs was more holistic, rationalises workload and is more representative of the real world (Table 10). However, the whole process of using the CiP method of assessment is dependent on trainees and trainers having knowledge of the process and how to complete CiP study forms. Importantly, there was feedback from the evaluation that training is required in order for trainees and trainers to become familiar with the CiP method of assessment.
In terms of time commitment (Table 4) in completing CiP documentation, there was considerable variation in opinion. Ten educational supervisors felt it took less time to complete and four educational supervisors felt it took more time to complete. In contrast, the majority of trainees and clinical supervisors felt the CiP forms took the same or more time to complete. This is encouraging for the first use of a new process, however the suggestion that some trainees and clinical supervisors feel the CiP forms are more time consuming could risk non-engagement in the process.
When considering fairness (Table 5), the majority of participants (44 people) felt that it had the same fairness with 31 participants feeling that it was more fair. Only one educational supervisor felt it was a less fair process.
When asked about the level of supervision and if they felt there was a clear difference between each level of supervision (Table 6), 56 respondents felt that there was a clear difference whereas 24 did not. There seemed to be a larger proportion of trainees who felt there was not a clear difference between each level in comparison to supervisors. The numbers of levels (Table 7) of supervision were thought to be adequate. The majority of educational supervisors in a range of specialties and training levels were able to make entrustment decisions (Table 8).
We were also able to learn from the problems that arose from the study (Table 11). Firstly, it was difficult for trainers to make a single entrustment decision on a number of different clinical procedures in one CiP. Secondly, the non-clinical CiPs (CiPs 10–14) were more difficult to gather evidence for and more difficult to assess using an outcomes-based assessment. Thirdly, the levels of supervision described in Tables 2 and 3 were not clear enough so were causing confusion, as the expectation for each level was not descriptive enough.
In order to tackle some of these issues, the levels of supervision were reviewed and new level descriptors (described in Table 12) were piloted in order to assess whether they could be used as entrustment level descriptors. Participants felt that these levels of supervision allowed progression of the trainee to be monitored and gave feedback on the competency level of the trainee. The majority felt that they could now differentiate between the levels of supervision and that they were much clearer to use than the levels of supervision used in the PoC study. These new entrustment levels now form the basis for the GMC approved internal medicine curriculum.21
Another aspect that was explored in this focus group was to further assess whether the non-clinical CiPs could be assessed using these entrustment level descriptors. The PoC study revealed that supervisors found it difficult to assess the non-clinical CiPs using the entrustment level descriptors. This view was reiterated in the evaluation from the development day focus workshop. The majority of participants felt that it was difficult to assess the non-clinical CiPs using entrustment levels. As this was a recurring theme it was decided by the Internal Medicine Committee that for non-clinical CiPs assessing some of the GPCs would be measured using anchor statements rather than entrustment levels.
The GMC has published guidance on GPCs through the GPC framework.2 The non-clinical CiPs in the internal medicine curriculum have been modified in order to reflect the GPC framework. There will be six generic CiPs, outlined in Box 1.
Generic capabilities in practice.
Able to successfully function within NHS organisational and management systems.
Able to deal with ethical and legal issues related to clinical practice.
Communicates effectively and is able to share decision making, while maintaining appropriate situational awareness, professional behaviour and professional judgement.
Is focused on patient safety and delivers effective quality improvement in patient care.
Carrying out research and managing data appropriately.
Acting as a clinical teacher and clinical supervisor.
Each generic CiP will be assessed using global anchor statements rather than entrustment levels.
Below expectations for this year of training; may not meet the requirements for critical progression point.
Meeting expectations for this year of training; expected to progress to next stage of training
Above expectations for this year of training; expected to progress to next stage of training.
Eight specialty or clinical CiPs will describe the clinical tasks essential to internal medicine (Box 2). These will be assessed using the new entrustment level descriptors in Table 12.
Specialty capabilities in practice.
Managing an acute unselected take.
Managing an acute specialty-related take.
Providing continuity of care to medical inpatients, including management of comorbidities and cognitive impairment.
Managing patients in an outpatient clinic, ambulatory or community setting, including management of long-term conditions.
Managing medical problems in patients in other specialties and special cases.
Managing a multidisciplinary team including effective discharge planning.
Delivering effective resuscitation and managing the acutely deteriorating patient.
Managing end-of-life and palliative care skills.
Our study found that supervisors perceived that they could not assess a multitude of different clinical procedures of differing complexity using a single CiP. Available options were to use a single CiP for each procedure or to remove procedure assessment from entrustment levels. Using a single CiP would result in the addition of 10 CiPs to cover each procedure, increasing the supervisory burden of assessment. Following a wider consultation, it was decided that procedures would continue to be assessed using direct observation of procedural skills assessment forms.23
This work identified a need for both supervisors and trainees to receive instruction in the new assessment method prior to implementation of the internal medicine curriculum, as the process designed is very different to the current competency-based model.
The study has a number of limitations: participants were a self-selected group of both trainers and trainees, who were a motivated group, more likely to engage in a new process, likely resulting in bias in response to survey questions. In addition, only 78 participants completed evaluation forms, a small sample who may not be representative across the cohort.
Conclusions
Our PoC study suggests that outcomes-based assessment is more authentic for educators and allows supervisors to make a more holistic assessment of their trainee. Educational and clinical supervisors as well as trainees will need to be trained in using CiPs prior to implementation and this process has now commenced from August 2019 for the implementation of internal medicine training.
Supplementary material
Additional supplementary material may be found in the online version of this article at www.rcpjournals.org :
S1 – Specialties invited to the proof of concept study.
S2 – CiP Study forms.
S3 – Evaluation forms.
S4 – Stakeholders.
- © Royal College of Physicians 2019. All rights reserved.
References
- ↵
- Shape of Training
- ↵
- General Medical Council
- ↵
- General Medical Council
- ↵
- Black D
- ↵
- Tham T
- ↵
- Kramer A
- ↵
- Black D
- ↵
- ↵
- Ten Cate O
- ↵
- van der Vleutun CP
- ↵
- Leung WC
- ↵
- Hawkins R
- ↵
- ↵
- ↵
- Ten Cate O
- ↵
- ↵
- Van Loon KA
- ↵
- Ten Cate O
- ↵
- Joint Royal Colleges of Physicians Training Board
- ↵
- Black D
- ↵
- Association of American Medical Colleges
- ↵
- Hall Render
- ↵
- Joint Royal Colleges of Physicians Training Board