ABSTRACT
The annual review of competence progression (ARCP) is a high-stakes assessment which all UK postgraduate trainees undertake to ensure competence progression. Previous evaluations of the effectiveness of the ARCP as an assessment have reported deficiencies in both validity and reliability, however, there has been little focus on the educational impact of the ARCP.
We conducted a mixed methods case study involving questionnaire, interviews and a focus group examining the impact of the ARCP on a respiratory higher specialist training programme. Participants included both trainers and trainees.
Perceptions of impact were mixed. The ARCP was reported to promote broad curriculum coverage, enable educational planning, provide educational governance and facilitate relationships with supervisors. However, participants reported that activities promoted by the ARCP may detract from learning and that issues of reliability and validity undermined the process. In some cases, this was reported to lead to disillusionment and stress for trainees. Concerns were raised that the process promoted a reductionist approach to education.
This research has resulted in several changes to local training, however, it has potential implications for the ARCP as a wider process. Trainers should be cognisant of the shortcomings of assessments and their impact on trainees, training and the future of the profession.
Introduction
The annual review of competence progression (ARCP) is ubiquitous across UK postgraduate medical education and is central to the outcomes-based structure of medical training.1,2 While not originally designed as a summative assessment process, the purpose of the ARCP is now defined by Health Education England (HEE) as a tool to ‘assess trainee achievement and learning, and suitability to progress to the next stage of training.’3 The ARCP is structured to provide consistency and objectivity, and to promote equity in assessment of postgraduate trainees.1,3
As an assessment, the ARCP has been subject to several concerns and criticisms. Many of these centre on validity, the performance of the ARCP in assessing trainees' achievement, learning and suitability to progress. Several studies have reported that trainees feel ARCP performance correlates poorly with clinical and professional competence.4–6 Reliability, the likelihood that the same trainee would receive the same outcome from a different ARCP panel, has also been recently highlighted as a concern.3,7
While validity and reliability have been studied, other criteria of good assessment (such as educational effect) or overall impact of the ARCP have received relatively little attention.8 Previous work focusing on other aspects of the ARCP have raised the possibility that the process may have unintended negative educational consequences.4–6
We sought to understand the impact of ARCP following local concerns about the effect on trainees and trainers working in respiratory higher specialty training (HST) in the Health Education England Yorkshire and Humber region (HEE-YH). Both trainees and trainers had expressed frustration at the ARCP process. Feedback to the ARCP panel was that the process was consuming a significant amount of trainee time in evidence collection and portfolio preparation. Trainees informally reported that they were seeking experiences which they judged to be of little educational value to satisfy ARCP requirements. There were reports of trainee stress as a result of the process, particularly around negative outcomes. Some supervisors had indicated that they were unsure as to the intended purpose of the ARCP.
ARCPs in respiratory medicine HST are undertaken remotely according to the process described in the Gold Guide, with outcomes defined by the specialty training curriculum.1 Within this document is an ARCP decision aid which is particularly pertinent to the ARCP process.1,9,10 This document outlines a list of minimum evidence requirements which must be presented by a trainee each year in order to succeed in their ARCP and progress unhindered to the next year of training. As such, it is a key part of the ARCP process, detailing expectations to trainees, trainers and the ARCP panel.
In this study, we investigated trainee and trainer perceptions of the impact of the ARCP in the context of respiratory HST in the HEE-YH region.
Methods
An exploratory sequential mixed methods case study was undertaken (Fig 1). As the aim was to understand the impact of personal experience, a constructivist paradigm, whereby findings emerge from reporting of personal experience, was assumed.11
Study design and phases.
Questionnaire: A link to an online anonymous questionnaire was disseminated via email invitation to all trainees in respiratory medicine in HEE-YH in spring 2018 (supplementary material S1). Trainees were identified from a list provided by the Specialty Training Committee (STC). Forty-two of 60 invited trainees completed the survey. The initial aim of the questionnaire was to confirm and clarify the nature of the concerns raised, to highlight where the ARCP was having a positive impact, to identify potential local solutions to any negative impacts and to feed these back to the STC.
Following an analysis of the questionnaire responses, a literature review of the ARCP process was conducted. This confirmed that most of the concerns highlighted were not isolated local issues but mirrored the experience of ARCP reported in other regions and training programmes. The results of the literature review were used in conjunction with questionnaire responses to highlight topics meriting in-depth exploration and further clarification in the second phase of the study. The second phase of the study aimed to provide an in-depth analysis of the impact of the ARCP in the context of a regional HST programme.
Interviews: Email invitations were disseminated to all respiratory HST educational supervisors (ESs) in the HEE-YH region. These were identified from a list kept by the STC. Respondents were recruited to attend a face-to-face in-depth semi-structured interview during summer 2019. A detailed guide to the focus of the interviews is included in the supplementary material S1. Interviews were recorded and transcribed verbatim with preliminary data analysis occurring through the interview period, allowing areas of interest and uncertainty to be identified and explored in increasing depth in subsequent interviews as a part of an iterative process. Five interviews were conducted with participants from different hospitals and a spectrum of training roles (including training programme directors and those with and without ARCP panel experience).
Focus group: All respiratory HST trainees in the HEE-YH region were identified by the STC and invited by email to a focus group held in summer 2019. All trainees expressing interest were recruited to participate in the focus group. This explored how participants' experience of ARCPs influenced their perceptions of the ARCP purpose and the impact of the ARCP. The focus group was recorded and transcribed verbatim. Eight trainees participated with representation from specialty training years (ST) 3 to 7. The focus group guide is included in the supplementary material S1.
Data collection was conducted by an insider researcher who, at the time of the questionnaires, was trainee representative to the STC.12 After seeking ethics committee advice, phase 2 data collection was designed to coincide with the period of time when this researcher had completed all ARCPs and was, therefore, no longer subject to the process, but was still a trainee so had no formal supervisory role with respect to trainee participants. In order to ensure objectivity, the co-researcher had no formal role in or prior knowledge of the training programme.
Quantitative data from the questionnaire was processed using MS Excel. Qualitative data from questionnaire responses was combined with transcripts from interviews and focus groups prior to analysis. Data analysis was conducted using thematic analysis.13 Further details of the data analysis process are provided in supplementary material S1.
Ethical approval was granted by the University of Leeds School of Medicine Research Ethics Committee and Health Education England Research and Innovation Committee. All participants provided written consent.
Results
Three distinct but related themes emerged:
the impact of ARCP decision aid
the impact of portfolio assessment
the ES role.
These were used to create a schematic representation of the process experienced by participants (Fig 2). Findings refer to qualitative results except where numerical values are explicitly stated.
Main findings. ARCP = annual review of competence progression; ES = educational supervisor.
The impact of the ARCP decision aid
The ARCP decision aid was seen as fundamental to ARCP preparation with over 90% of trainees utilising this resource.10
The decision aid was described by all participants as clearly communicating requirements and facilitating assessment-driven learning. It was felt to focus learning to the curriculum and help to identify gaps in skills and knowledge, enabling proactive educational planning and breadth of curriculum coverage. Trainer 1:
I'm able to focus their minds earlier on, through the year, to say look, this is what you need to do, look at your curriculum and get this signed off before ARCP.
While several trainees reported that collection of evidence facilitated reflective learning, participants also reported that the ARCP decision aid promoted other behaviours and activities distinct from learning, termed assessment driven performance.14 Examples included:
collection of evidence of experience rather than learning
engagement in activities perceived to be of no educational benefit
diversion of training time to prepare portfolios
active avoidance of formative feedback opportunities lest these be perceived negatively at the ARCP.
Furthermore, concerns were expressed that using a decision aid to prepare for the ARCP may promote a reductionist approach to learning and training. All participants were aware of the concept of a ‘tick box’ ARCP. Most trainees felt that the ARCP decision aid represented the minimum criteria to progress through training and it was therefore reasonable to approach the ARCP by aspiring to minimum criteria, however disempowering this may be. Trainee 2:
I've got a hoop to jump through. They've defined the hoop and I shall jump.
Trainees simultaneously expressed disillusionment resulting from the minimum standards approach and requirement for assessment driven performance as opposed to learning. Trainee 21 (questionnaire):
The process is not set up to recognise achievements. Myself and my colleagues are doing interesting research, educational activities, master's degrees ... all you get from ARCP is ‘you need a knee aspiration DOPS’.
Some trainers went further implying that the reductionist attitude towards training arising from ARCP may entrench itself within the profession. Trainer 2 (ARCP assessor):
We almost aim towards presenting the minimum level of evidence across the board for most people and that starts to become acceptable.
The impact of portfolio assessment
The key concept underlying this theme is the impact of the ARCP as an indirect assessment of competence. Less than 50% of trainees in the questionnaire agreed that their experience matched HEE aspirations for ARCP to ‘assess achievement, learning and suitability to progress in training (Table 1).3 The validity of the ARCP as an assessment was reported as a key concern. Trainee 32 (questionnaire):
Trainee questionnaire responses to statements about the annual review of competence progression
It is just a check-box exercise. It doesn't quite reflect the quality of trainees clinically.
All participants reported the concept of a ‘good trainee’. While the aspiration of ARCP may be to identify the ‘good trainee’, most reported this as distinct from the trainee who excels at ARCP. Conversely, several participants reported that they felt an ARCP would not necessarily identify the ‘bad trainee’. One trainer highlighted that the distinction between ARCP competence and clinical competence may redefine the concept of the ‘good trainee’. Trainer 2 (ARCP assessor):
Is a good trainee someone who fulfils all the requirements of ARCP for that year and does nothing else? Or is the good trainee the one who has a whole host of diverse interests ... but they don't fulfil the criteria for ARCP. Which of those is the good trainee? It depends on how you define what a good trainee is.
Several trainees reported difficulty in using the e-portfolio as exacerbating difficulty with ARCPs and less than 20% trainees agreed that the process was consistent (Table 1). Trainee 10 (questionnaire):
I am aware that I passed some early ARCPs that my colleagues failed even though I lacked the same evidence.
Overall, the limitations of the ARCP were reported to undermine the whole assessment process with some participants expressing disillusionment with the process. Several trainees reported stress arising from the ARCP. For some, the ARCP was regarded as a redundant assessment with a hidden curriculum of competences and achievements beyond the ARCP eluded to. Trainee 7:
I guess [the ARCP] identifies the minimum but it doesn't help you to get a consultant job at the end – that's all the other things you have to find time for.
The educational supervisor role
This theme encapsulates the educational impact of the interplay between ESs and the ARCP.
All participants were clear that the ES was key to determining a trainee's success at ARCP. Over 70% of trainees in the questionnaire found that their ES was a useful source of information about ARCP preparation. However, 16% of trainees reported that their ES lacked the time or training required (Table 1). Several ESs and trainees recognised that the ARCP has a role in educational governance and assessment of supervision quality.
Most participants agreed that the ES report was the most important piece of evidence presented for ARCP. A comprehensive and complimentary ES report was recognised to mitigate other portfolio deficiencies which may be more easily identified if the ES report was sub-standard. Trainees expressed that a direct longitudinal assessment in the form of the ES report had potential to increase ARCP validity. It was considered that, in practice, the assessment of competence is performed by the ES rather than the ARCP panel. Trainee 3:
They are the people who see you and work with you, whereas the ARCP are looking from afar, so it's reassuring to know that someone who's worked with you is happy.
However, concerns were raised that the process risked becoming an assessment of the supervisor rather than the trainee and that a heavy reliance on the ES report risked exacerbating the influence of ES bias, detracting from ARCP reliability.
Many participants recognised that notions of competence were difficult to define within the curriculum and, as such, the ES was often responsible for judging and reporting on competence. Trainer 4 (ARCP assessor):
There is an element of judgement and trainees will always have to accept that. Being a consultant is a complicated job and to reduce it to a shortlist of things is clearly not going to adequately prepare people for the role ... there's always going to be a degree of interpretation.
Discussion
This study found that the experienced impact of the ARCP in respiratory HST in the HEE-YH region was wide ranging with both positive and negative consequences. The availability of curriculum documents and the relationship with the ES promoted by the ARCP experience was perceived to positively impact training behaviours (such as proactive educational planning, comprehensive curriculum coverage and reflective learning). However, unintended consequences of the ARCP were reported to divert from training, undermine assessment outcomes and result in a reductionist approach to education. Many participants raised issues around the reliability and validity of ARCP assessment resulting in disillusionment with potential to shift future professional values and aspirations. The impact reported by participants appeared to be predominantly the result of national ARCP processes rather than local implementation.
Many of the findings presented echo those described in the wider literature around outcomes-based education and assessment. The General Medical Council promote outcomes based curricula for numerous reasons, including public protection and uniformity of training.2,15 One of the key ways in which outcomes are communicated to learners in any educational programme is via assessment.16 The ARCP decision aid lists assessment requirements, thereby conveying the expectations of training. Participants reported this to be beneficial, ensuring broad curriculum coverage and promoting self-directed learning. However, in line with previous research, some ARCP and curriculum requirements were felt to be, at best, of questionable educational value and, at worst, perpetuating poor practice.4 This concern was succinctly summarised by one trainer regarding the requirement to produce evidence of experience (eg logbooks or clinic attendance). Trainer 2 (ARCP assessor):
To have experience doesn't mean that you learn from that experience or that you learn appropriately from that experience.
It is clear that the ARCP decision aid is key to informing trainees of expectations, yet careful consideration should be taken when considering the message conveyed by this document. By emphasising minimum numbers of assessments and experience, rather than quality of assessments and achievements, the ARCP risks misrepresenting the expectations of HST, shifting the focus from quality of learning to quantification of evidence.14
The majority of participants in the study expressed concerns that the efficacy of ARCP as an assessment tool undermined the entire process. This corroborates the findings of previous studies of the ARCP.4,5,7 Key issues raised were reliability and validity. A recent HEE review of postgraduate assessment recommended improvements in consistency and training of ARCP panels in order to improve reliability of assessment.3,7 While this will be a step in the right direction, participants in the current study eluded to other issues including variation in quality of supervisor reports, nuances in interpretation of competency and difficulties in using the eportfolio as contributing to reliability issues. It seems unlikely that these issues would be isolated to respiratory HST in HEE-YH. The central role of the ES in making a judgement about professional competence has significant implications for reliability given that no two supervisors will have identical expectations, experiences and skills. Much of the data presented here was collected prior to the implementation of the HEE recommendations and, as such, it has yet to be seen whether these will impact trainee experience.
Validity was a fundamental concern for trainees and supervisors, leading to a perception of the ARCP as an unfair assessment that may fail to achieve its aims, particularly differentiation between the ‘good trainee’ and ‘bad trainee’. Issues of validity have been highlighted in other studies of ARCP.4,6,7 Several participants expressed that this undermined the assessment, potentiating the risk of a pervasive reductionist educational culture developing. For other trainees the limitations of the ARCP resulted in stress and anxiety.
Some participants described that the ARCP as an assessment was so undermined that success was distinct from the qualities required for consultant level practice. A hidden curriculum was eluded to within which some ARCP requirements were perceived as irrelevant to practice as a respiratory medicine consultant (eg knee aspiration), while achievements above and beyond ARCP requirements were either expected or desirable (eg significant research experience and postgraduate qualifications).17 Further work is needed to understand this hidden curriculum, to assess whether this is specific to the context of this case study or more widespread, and to appreciate how a hidden curriculum may be consolidated within the ARCP process. The existence of a hidden curriculum may counter concerns about educational reductionism.
The observation that individual ES judgements are often the key factor underlying an ARCP outcome raises several issues. Several participants described that, in essence, the ARCP process might be considered as quality assurance process overseeing the ES assessment. In the event of a suboptimal ES report, multiple trainers on the ARCP panel would be asked to make individual judgements about competence. This occurs in the absence of an explicit documented definition of competence in many areas, raising concern in the context of an objective ARCP process.3 The lack of explicit definitions of competence is counter to aspirations for reliability. However, it may not be possible to make a comprehensive explicit statement of outcome requirements for a 5-year higher specialty training programme. Medical training since the days of Hippocrates has relied on the professional judgement of trainers to assess trainees. In recent times, there has been a drive to increase objectivity yet, in reality, there may be limits to the extent to which true objectification in assessment can be achieved.18 Once again, these issues are unlikely to be confined to the context of this case study. Within physician training, there are moves to address these issues by transitioning to generic capabilities rather than specific competences in an effort to bring ARCP judgements in line with the realities of clinical practice.19,20 Future iterations of HST curricula are planned to be structured along similar lines. While this does not in any way remove the requirement for trainers to exercise professional judgement, it sanctions this as an open process within training.
Limitations
We have described a case study based on a single specialty within a defined region of the country. While our results correlate with those of other groups, caution should be taken in directly applying these to future cohorts or settings. The study was undertaken using a constructivist paradigm and, as such, the findings reported are representative of participant experience. We do not describe an objective positivist overview of the ARCP.
The research was performed over a period of 18 months during which time a local educational programme was implemented to raise awareness of ARCP process including enhanced local induction and instigation of ARCP awareness sessions as part of the regional teaching programme. The HEE review of the ARCP process was also published during the period of data collection. An improvement in ARCP outcomes has been observed locally over the 2 years since the initial questionnaire. All of these factors may have influenced data obtained from trainees in the latter phase of the study.
We acknowledge that the role of the insider researcher is likely to have influenced data.12 Insider research is a recognised method within qualitative research with benefits including enhanced appreciation of the research context, facilitating depth of exploration of the subject and identification of key issues. The drawbacks of this method include the impact of preconceived perspectives regarding outcomes and potential influences on participant responses which are impossible to fully assess. We have attempted to mitigate these drawbacks where possible by inclusion of a co-researcher with no prior knowledge of the participants or context, incorporating anonymised responses to phase 1 data collection and by timing the second phase of data collection to minimise the impact of power dynamics on participants.
Implications
To our knowledge, this is the largest study of the impact of the ARCP in medical higher specialty training. The study achieved breadth by incorporating the views of multiple stakeholders including over two-thirds of eligible trainees. It is also the first study of ARCPs to assume a constructivist methodology thereby ensuring depth in the comprehensive appreciation of the experience of participants.
The findings of the current study have been used to guide the approach of the local STC to the ARCP process. Induction for new trainees now incorporates a dedicated section on ARCP approach and structured format for presentation of evidence has been promoted (Box 1). Existing trainees have regular updates on ARCP preparation. Every ES within the region is now given individualised feedback on their reports from the ARCP panel. However, the ARCP impact appears to predominantly emerge as a result of the nationally defined process rather than local implementation of the process. From a wider perspective, we consider that it is important to openly acknowledge mounting evidence around the limited efficacy of the ARCP as a summative assessment and the impact of this on trainees. We should be as honest about the limitations of our educational tools as we would be expected to be about the limitations of our clinical tools in order to reduce the propensity for trainee disempowerment and disillusionment. Further work is needed to understand the role of the hidden curriculum achieving consultant-level practice and in translating this to ARCP assessment.
Implications of this work
Summary
In this study, we found that participants' perceptions of the impact of the ARCP in the context of respiratory HST in the Yorkshire and Humber region were mixed. The ARCP exerted a clear positive influence by promoting curriculum coverage and facilitating a structured approach to the training programme via assessment driven learning. However, in line with previous work, we found that the ARCP was perceived as a flawed assessment, limited in both validity and reliability. The impact of this was to disillusion and disempower trainees, to undermine the assessment process and to risk driving down professional aspirations and altering the future values of the profession.
Supplementary material
Additional supplementary material may be found in the online version of this article at www.rcpjournals.org/clinmedicine:
S1 – Study design, interview guide and thematic analysis process.
Acknowledgements
We would like to thank the Yorkshire and Humber Respiratory Specialty Training Committee for supporting this research.
- © Royal College of Physicians 2021. All rights reserved.
Jump to section
Related Articles
Cited By...
- No citing articles found.