The independent validation of the Foundation Programme application process: a closer look

Editor – We highlight four clarifications to Harris and colleagues’ comments about the validity study for the situational judgement test (SJT) for Foundation Programme recruitment.¹

Sampling technique and size

Harris et al¹ question the 47.7% response; however; this is typical of peer-reviewed published research.² They also question the sample size, yet n=391 is acceptable for a preliminary study, exceeding many other SJT validation studies;³ it was not practical to obtain whole cohort data. Harris et al query the remedial action sample, which is, by nature, very small; yet we explicitly stated that this should be interpreted cautiously, with further research required.

We agree that a sampling approach in future studies should examine outcomes across whole or several cohorts of foundation year 1 (FY1) doctors, if logistics allow.

Validity of supervisor ratings

Selection research typically uses supervisor performance ratings as outcome data. Harris et al suggest supervisors may not be familiar enough with FY1 doctors to make accurate judgements. However, the overwhelming majority (83.6%) of supervisors were confident in their judgement.

Harris et al question using internal consistency (Cronbach’s alpha), yet this is widely used in questionnaire evaluation.⁴ Harris et al also suggest items were not independent: two items ‘seem to be testing the same construct’. However, these two items assess behaviours under ‘problem solving and decision making’, so intentionally target the same construct.

Integrity of the study

Harris et al ask ‘should entities who implement new selection tests be responsible for validating them’ and state ‘this work did not undergo external academic peer review’. These statements are inaccurate and misleading. Our study was subject to peer review at every stage of its design, implementation and analysis; overseen by Health Education England, including representatives from the UK Foundation Programme, British Medical Association, Medical Schools Council, General Medical Council, NHS employers, medical and foundation schools, and an independent academic was commissioned to review the report.

Test developers typically publish their own validation studies in addition to inviting external validation studies.

We support Harris et al’s invitation for independent research to examine the validity further. We agree that ongoing research and peer scrutiny is vital to inform continuous improvements.

What does the SJT measure?

It is implied that the SJT may not test relevant constructs. Yet, the study clearly demonstrates the SJT measures key competencies identified in the person specification. There also exists substantial evidence for construct validity of SJTs in many other settings.^5,6

Overall, we continue to welcome independent research and peer review to further evaluate the FY1 SJT and we are committed to ongoing improvements.

Conflicts of interest

KW is affiliated with the UK Foundation Programme Office, which oversees the delivery and scoring of the UK Foundation Programme SJT. FP is affiliated with Work Psychology Group, which provides consulting advice and support to Health Education England on selection methodology. Work Psychology Group does not receive any royalties for any methodology used.