The Situational Judgement Test: not the right answer for UK Foundation Programme Allocation
Editor – Allocation to UK Foundation Programme training posts has long been an issue of contention. Following recent analysis of the allocation process, the UK Foundation Programme Office has now moved away from the use of a Situational Judgement Test (SJT) and the Educational Performance Measure (EPM). Although SJTs have been proposed as a suitable tool for aiding selection decisions, our article last year demonstrated that this was not the case when considering allocation of places in the UK Foundation Programme.1
We were interested therefore to hear of Sahota and colleagues' strong defence of the SJT2 and have carefully considered the points they raise. However, we remain unpersuaded by the arguments made – indeed we have discussed the issues with a wider group of senior academics from UK medical schools (several of whom are additional signatories to this letter), and it is our continued belief that the SJT was not an appropriate method of determining Foundation Programme allocation. To further illustrate this, we provide additional examples of the potential consequences of how an individual candidate's allocation could be dramatically impacted by factors outside of their control.
We estimate that the standard error of measurement (SEM) of the SJT is approximately 16 SJT points, or around 2.5 points of a student's total ranking score out of 100 (0.38 standard deviations). For a student with a ranking score at the mean, changing their SJT score by 1 SEM would change their position in the ranking by around 1,200 places (out of 8,000 or so applicants). This could inevitably impact their destination for the Foundation Programme – by chance alone. This effect is before we consider any unreliability due to issues related to a lack of concordance among the experts who determine the ‘correct’ response keys. The SJT Technical Report for 2021/22 notes that a Kendall's W of 0.5 was considered satisfactory.3 This is equivalent to just 50% agreement and below the 0.6 required for good agreement. Of course, some items will have higher agreement rates, but clearly there is doubt over the best course of action for some scenarios and importantly, this is recognised by the candidates, which in turn can lead to low confidence in the fairness of the assessment.
Sahota et al dispute that the SJT disadvantages Black and Minority Ethnic (BAME) students. However, SJT technical reports3 consistently state a difference in mean scores between BAME and white students of around 20 points, which equates to approximately 3 points in a student's total ranking score. Here we find an even bigger impact on the average student's ranking than with 1 SEM of SJT scores – approximately 1,500 places. Why BAME students do not perform as well as their white peers on the SJT is unclear, but this level of impact is not acceptable given its effect on allocation.
Sahota et al also question the statistical power and the interpretation of multivariate statistics in relation to the predictive utility of the SJT and EPM for disciplinary actions.4 However, the large sample size meant the study had acceptable power despite the overall risk of disciplinary action being low. The multivariate analysis in this paper demonstrated that the SJT had no incremental validity over the EPM and thus the significant incremental expense incurred by its use was not warranted for the outcome of predicting/reducing disciplinary action.
We demonstrated that the EPM was also an inappropriate tool for Foundation Programme allocation. Undergraduate medical assessments are not designed to stratify students across deciles, but to ensure that all graduates meet the required competencies to practise medicine safely as a Foundation doctor. Sahota et al are incorrect to state that the EPM is worth more than 9 points. In 2023 all students were awarded 41 points at baseline with a further 0 to 9 points added based on decile rankings. For each decile increment, students gained one extra point out of 100 in the allocation ranking. These decile rankings drove highly competitive and undesirable behaviours among medical students throughout their programmes.
The unnecessary pressure and stress experienced by medical students trying to improve their decile score as well as preparing for and sitting the SJT was universally apparent. The announcement of the upcoming removal of both the SJT and EPM has been welcomed by the majority of the student body nationally and early indicators are that this will have a significant positive impact on student welfare and experience.
As all UK medical graduates are required to complete the Foundation Programme, determination of graduate placements has never been an issue of personnel selection – it is one of allocation. We have demonstrated that neither the SJT nor the EPM, in isolation or combined, is fit for purpose in allocating students to Foundation placements and were widely regarded as unfair by students and unhelpful by medical schools. This is perhaps best summarised by the view of many students that ‘the EPM is the Hunger Games and the SJT is the randomiser’.
- © Royal College of Physicians 2023. All rights reserved.
References
- ↵
- Sam AH
- ↵
- Sahota G
- ↵
- Washbrook M
- ↵
- Sam AH
Article Tools
Citation Manager Formats
Jump to section
Related Articles
- No related articles found.
Cited By...
- No citing articles found.