The independent validation of the Foundation Programme application process: a closer look

The UK Foundation Programme Office recently announced that the situational judgement test (SJT) and education performance measure (EPM) should continue to be used for Foundation Programme selection, as supported by an independent study.^1,2

This study collected data on a small sample of around 4.8% of applicants to the UK Foundation Programme in 2013 (391/8,162). Supervisors in five foundation schools across the UK completed questionnaires assessing foundation year 1 doctors (FY1s) in the domains argued to be measured by the SJT. The questionnaire comprised a six-point scale ranging from (1) ‘needs significant development’ to (6) ‘clear area of strength’ with a freehand section for additional comments. Data were collected on whether FY1s passed their annual review of competency progression assessment (ARCP) and whether the supervisor was aware of remedial action being instituted for a particular doctor.

The study concluded that higher SJT scores were associated with higher ratings of FY1 performance; that the two selection tools (SJT and EPM) are complementary in providing prediction of performance, and that FY1 doctors in the low scoring SJT category were almost five times more likely to receive remediation than those who were in the high scoring category (6.1 and 1.3% respectively).

A closer look at the data in the report revealed that over half of the questionnaires were not returned (52.3%). Most of the questionnaires were filled in by educational supervisors (48%, 187/391) and foundation programme directors (FPDs) (43%, 169/391). FPDs and some educational supervisors rarely see their trainees in clinical situations, and often rely on the NHS e-portfolio to get information on their trainees. There are several potentially useful sources of information on the e-portfolio to help educational supervisors and FPDs rate trainees. However, the domains assessed in the e-portfolio are not aligned with those on the study questionnaires or the SJT. Although there are overlaps in the domains, such as commitment to professionalism and working effectively as part of a team, there is not necessarily enough information to answer a questionnaire with detailed breakdowns of each section. There is no compulsory recording of the ability of the FY1 within the domains of ‘coping with pressure’ or ‘patient focus’. This is potentially why the supervisor confidence in rating performance of individual FY1s was low. Only 25% of the supervisors gave themselves the highest accuracy rating and many supervisors used either one or two ratings across the entire questionnaire (57%), for example selecting a score of four to rate every domain.

Furthermore, the use of Cronbach's alpha for calculation of the internal consistency of the questionnaire seems inappropriate. The questionnaire items do not appear independent of each other. For instance, ‘Made appropriate decisions based on relevant information’ and ‘Sought to understand and solve problems by taking into account the whole picture’ seem to be testing the same construct.

The conclusions on the validity of the Foundation Programme selection tools from this study are based on a small sample. The findings related to the SJT score and those needing remediation are based on only 16 candidates.

Moreover, the study was carried out by the Work Psychology Group who were involved in both the development and implementation of the Foundation Programme SJT.³ Perhaps, other groups would be better placed to carry out a fully independent review, without a potential conflict of interest. In addition, despite the potential impact of the reports on UK medical selection, this work did not undergo external academic peer review.

Using the current method, the most relevant information is to be gleaned from clinical supervisors who see FY1s regularly across a range of clinical situations; they tend to be the consultant on the ward on which the trainee is working. An alternative approach could be to examine SJT scores and ARCP outcome across a whole or several cohorts of UK foundation trainees.

Finally, this study does not address the pivotal question: Do SJTs actually test the constructs they intend to? As well as raising the question, should entities who implement new selection tests be responsible for validating them? We encourage debate and independent research in these key areas.