Time for preference-informed foundation allocation?
ABSTRACT
Successful completion of year 1 of the UK Foundation Programme is a General Medical Council requirement that newly qualified doctors must achieve in order to gain full registration for licence to practise in the UK. We present compelling evidence that both sections of the UK Foundation Programme allocation process, consisting of the Educational Performance Measure and Situational Judgement Test scores, are not fit for purpose. The ranking process drives competitive behaviours among medical students and undermines NHS teamworking values. Furthermore, data from 2013–2020 show that UK minority ethnic students consistently receive significantly lower SJT scores than White students. The current process in the UK allocates lower ranked students, who often need more academic and social support, to undersubscribed regions. This can lead to vacancies in less popular regions, ultimately worsening health inequality. A preference-informed allocation process will improve trainee access to support and help retain trainees in underserved regions. We aim to summarise the flaws of the current system and report a potential radical solution.
The UK Foundation Programme is a fundamental component of postgraduate medical training for UK medical school graduates joining the NHS. Successful completion of the first year of the UK Foundation Programme (UKFP) is a General Medical Council (GMC) requirement that newly qualified doctors must achieve in order to gain full registration for a licence to practise in the UK.
Since 2011, the UK Foundation Programme Office (UKFPO) has used a ranking system based on scores from two metrics (the Educational Performance Measure (EPM) and the Situational Judgement Test (SJT)), each contributing to 50% of the candidates’ final scores. UKFP placements are then allocated to medical graduates based on their submitted location preferences, starting with the highest ranked candidate. As UKFP posts fill across the UK, lower ranked candidates, who may need more support based on their academic achievements and situational awareness, are more likely to be allocated to locations that are not their first preference. Since 2014, fewer than 79% of applicants obtained their top UKFP preference.1 Allocation to a UKFP post that a candidate did not seek may lead to potential adverse consequences for the individual and their patients. It has been suggested that the current system, in the long term, can exacerbate health inequalities.2
From the traditional self-organised, direct applications for individual jobs to a national process using white space questions in 2006 before adopting the current combined EPM and SJT ranking system, access to the UKFP has undergone several iterative changes over the years. However, despite ongoing efforts to increase transparency and fairness within the application process, there is a broad consensus that the current combined EPM and SJT system has significant flaws.3
The EPM, which provides a score out of 50, is calculated based on a student's decile ranking at their own medical school. The challenge lies in the fact that institutions use a wide range of assessment methods for knowledge-based and clinical examinations with variable inclusion of early years results. The use of common content assessment items in knowledge tests across different medical schools has demonstrated the wide variation in results, ranging from 5% to 40% in failure rates on a standard set of knowledge-based questions if the same pass mark is adopted.4 This calls into question whether decile rankings allocated within medicals schools can be compared between medical schools.
Although the EPM is scored out of 50, the range of marks between lowest and highest decile is only nine points (the candidate in the lowest decile is awarded 34 points, and the highest 43 points; Fig 1). The decile score, thus, has a very small impact on the overall EPM but, for the duration of undergraduate medical education, it is the only factor that students can modify to improve their prospects of gaining their first choice UKFP allocation. However, it should also be noted that there is evidence demonstrating a negative effect of non-White ethnicity in undergraduate assessment outcomes that impacts the EPM.5 Seven points were previously awarded for educational achievements (such as additional degrees and publications), but these are due to be withdrawn in 2023 due to concerns about equity of access for students from disadvantaged backgrounds.
As higher scoring candidates are deemed to be more likely to secure their preferred UKFP placements, the EPM is a strong driver for competitive behaviours in medical school, which is in direct conflict with the teamworking values of the NHS constitution.6 Medical schools and students have become increasingly concerned about the adverse negative impact and unnecessary anxiety created by the EPM on the culture of medical schools and behaviour among medical students.
The SJT aims to examine non-academic attributes, such as patient focus, commitment to professionalism, coping with pressure, effective communication and teamworking. The SJT comprises workplace scenarios where the student considers themselves to be a foundation year-1 doctor and answers rating, ranking and multiple-choice questions based on what they should do (not what they would do) in the given scenario.
The SJT has come under criticism as to whether it is a valid and acceptable assessment tool for UKFP allocation.3,7 There are inconsistencies among experts developing the SJT, with a significant number of subject matter experts failing to agree on the ‘correct’ answers.8,9 Furthermore, a random guess on a single SJT ranking question will score 12/20 on average, which equates to 1.85 points in a student's overall ranking due to the SJT scaled scoring system.3 With a difference of only nine points between the top and bottom deciles in the EPM, five random guesses (5 × 1.85 points) in the SJT assessment becomes equivalent to five years of undergraduate educational achievement (9 points).
Data from UKFP 2020 found that 82.5% of student SJT scores lie between 35 and 45 points, which does not permit any meaningful discrimination between candidates.10 Furthermore, SJT scores are not predictive of disciplinary action by the GMC, which raises the question of whether the SJT is serving its intended purpose in the UKFP application process.11 Moreover, the SJT is found to be consistently biased against UK minority ethnic students compared with UK White students, with reports from the UKFPO from 2013–2020 confirming that UK minority ethnic students receive significantly lower scores than their White counterparts.1 As a result of this bias, it is likely that minority ethnic applicants are significantly disadvantaged compared with White graduates in terms of achieving their desired allocation.
With the UK Medical Licensing Assessment due to commence in 2024/2025, in addition to the existing Prescribing Safety Assessment and internal medical school examinations, the assessment burden on students continues to increase. Examinations cause stress and significant demands on student time and energy, which is known to negatively impact wellbeing. Furthermore, additional assessments detract from other essential learning and preparedness for practice in the vital final year of medical school. With such potential negative impacts on fairness, working culture and student wellbeing, it is imperative to question the continued use of both the EPM and SJT as an appropriate tool for allocating graduates to UKFP placements. If the SJT has a role in identifying individuals who may benefit from additional education and support, then there would be a case for bringing the assessment forward in the undergraduate course, which would provide the opportunity for remediation rather than the potentially damaging impact on allocation in the UKFP.
As the UKFP is essential for full registration, all qualifying graduates must be allocated a place. The process is, therefore, one of allocation rather than selection, and all graduates should be entitled to a fair and transparent process. As the evidence demonstrates that the overriding factor influencing student placement preference is location, we propose using student preference alone for the purposes of UKFP allocation.12
Using student preference as the only variable for UKFP allocation would be a more equitable and defensible process than the current combined EPM and SJT system. The use of a ‘global happiness’ model will allow as many students to be allocated to their highest preference as possible. We propose the use of a preference-informed allocation system to achieve better, or at least similar, outcomes compared with the current ranking system.
The majority of students express preference for UKFP training closer to their family home.13 This is particularly significant for those who attended state-funded schools, are from non-White ethnic groups, or are from lower socio-economic groups.14 By facilitating the students to be allocated based on preference, the UKFPO will likely support NHS efforts to recruit trainees in underserved regions of the UK, which may help reduce national health inequality.
We propose that the use of preference-informed allocation will help eliminate unhelpful competitive cultures in medical schools, allow more fairness in the allocation process and provide a more even distribution of candidates across the UK, and ultimately benefit patient care.
Commentary
Prof Kamila Hawthorne, head of graduate entry medicine, Swansea University, Swansea, UK
The UKFPO have been running a scoring system of ranked candidates’ preferences since 2011 (based on EPM and the SJT) to allocate foundation places across the UK to newly graduated doctors.15 Many readers will remember the ‘old system’ whereby final-year medical students applied individually to favoured consultants for junior hospital officer posts, and the inherent inequalities in that system. Allowing new graduates to select their preferred place to work by a competitive ranking system sounds much more reasonable, but is it?
Even before we look at the psychometric characteristics of this system, as discussed in the paper by Sam et al, some perverse incentives stick out and are an issue for many who organise the learning of our medical students.
The ranking system encourages a fiercely competitive approach from medical students, who feel that they will be disadvantaged if other students rank higher than they do. In a learning system where we encourage teamworking, caring and empathy, this counter-intuitively becomes a cut-throat ‘dog-eat-dog’ system. As the head of the graduate entry medicine programme at Swansea University, I have had numerous representations from anxious and impassioned students about anything that is perceived as favouring subsets of students in their cohort. Issues range from a microphone that was not working properly in the exam venue, to deferrals due to being a COVID-19 case or contact, as students see their hard-won rankings threatened by circumstances outside their control. Keeping the peace and encouraging a collaborative approach can be challenging in these circumstances, and most other medical school education leads tell similar stories. Students are, by nature, competitive and a system such as this just lights the blue touch paper.
The paper by Sam et al is a summary of the predictive properties of the SJT and EPM ranking system. It relies on a well-referenced examination of the literature and the authors’ analysis of the process of calculating the ranking to make its points, rather than a detailed statistical analysis of its own. It does not instil confidence in the system. In essence, it suggests that the SJT, while being little better than a random test, is also discriminating in terms of differential attainment. The authors state that there is evidence of consistent bias (over a number of years’ data) against UK minority ethnic students. This is a grave concern and, from an assessment governance stance, should be urgently reviewed as the exam itself appears to be inherently unreliable. Surely it should be terminated immediately; what else should zero tolerance of racism mean? Many of us tolerate differential attainment between ethnic groups in undergraduate and postgraduate medical exams where necessary licensing standards need to be demonstrated to the public and try to find ways to mitigate its effects.16 But this is not a professionally necessary licensing exam, nor is it psychometrically rigorous.
According to Sam et al, the EPM also comes in for scrutiny, as it operates under different rules depending on the medical school and does not compare like with like across the UK. Amalgamating deciles across different schools into a universal scored ranking system is inherently unfair.
Sam et al don't stop here: in addition to an elegant exposition of the problem, they also suggest a solution. Their proposal needs careful modelling and consideration, alongside any other approaches that may be suggested; for example, using the incoming Medical Licensing Assessment results to determine selection for posts. In a complex situation, both suggestions will have inherent problems that will need teasing out. Surely it is time to show that we value the next generation of doctors sufficiently to offer them a fairer method of allocation to foundation posts, and practice what we preach in terms of educational and assessment rigour?
- © Royal College of Physicians 2022. All rights reserved.
References
- ↵
- UK Foundation Programme
- ↵
- Beck C
- ↵
- Harris BH
- ↵
- Hope D
- ↵
- Woolf K
- ↵
- NHS
- ↵
- Sharma N.
- ↵
- ↵
- Beesley R
- ↵
- Work Psychology Group
- ↵
- Sam AH
- ↵
- Miah S
- ↵
- ↵
- ↵
- ↵
- Esmail A
Article Tools
Citation Manager Formats
Jump to section
Related Articles
- No related articles found.