Abstract
In 2007, an assessment centre approach (a structured interview, a case-based discussion and a communication exercise) was implemented to replace the traditional interview for entry to core medical training. Feedback was obtained from 53 of 69 assessors, all consultants and most with extensive experience of the traditional system. Each station was rated by around 20 interviewers. This overwhelmingly rated the new process as useful in assessing the candidate (>90% for all stations). Comparison with the previous system was only provided by between 12 and 21 people per station. The structured interview was rated better (n=12), undecided (8), or worse (1); the case-based discussion better (16), or undecided (3); the communication station better (8), undecided (3), or worse (1). There is still work to do on the best components to include but the principle of multiple assessments to examine differing parts of the person specification seems, subjectively, to be supported.
Introduction
There are major and widely discussed problems with the current reforms in medical training being implemented under Modernising Medical Careers (MMC). However, there are elements of these reforms which might meet with the approval of physicians if they can be shown to be an improvement. One of these has been the move from traditional interview to an assessment centre approach. In the past most senior house officer (SHO) interviews would involve approximately 15–20 minutes of contact with the junior doctor and would concentrate on their previous experience, standard questions about audit, the management of a clinical problem and, possibly, a standardised exercise such as a two-minute presentation on why they should be selected. There was no regional or national consistency in approach and, in general, the method of selection and the standards were very much left to the interview panel to decide on the day. In 2002, the Chief Medical Officer identified deficiencies in SHO selection with lack of standardisation and failures to base selection on assessment of the competencies required to provide good quality care and to progress through training.1 A large body of work in relation to selection in many professions exists and its applicability to medicine is summarised in two recent documents.2,3 On the basis of expert opinion, MMC specified that an assessment centre approach must be used, including face-to-face contact for 30 minutes or more, but gave no further guidance about the content of the assessment. The person specifications for the posts became available in November 2006 and those organising the delivery of selection were instructed that these would inform the selection process. They were, however, left to themselves to determine how this might happen.
Methods
During the summer of 2006 a letter outlining the implications of the MMC reforms was circulated to all consultant physicians in the northern region of England. Those interested in helping with the reforms were invited to come forward. A working group (initially 10 people) was created from those who identified an interest in the development of interview tools and this met over the autumn of 2006. The group decided to plan three 12-minute stations in the assessment. There would be an interview, a case-based discussion (CBD) and a communication exercise. It was necessary that all areas of the person specification be covered and this informed the content of some part of the assessment. The structured interview involved six separate assessments, the communication station four, and the CBD three. To ensure that each of the three stations contributed a third of the total scores, the individual station scores were weighted.
The interview would enable the candidate to discuss their career aims, to demonstrate their commitment to the specialty and their understanding of audit and research. Discussion was followed by a question about a clinical mistake that they had made and one about how they would resolve a conflict about time pressures in getting to teaching when the wards were busy. All candidates were asked the same stem questions. Although some suggestions were made, follow-on questions were left to the discretion of the assessors.
The CBD would engage the candidate in a real life clinical case. For this assessment process 12 minutes of discussion were available. It was important that the case should involve some data interpretation and that it should also probe beyond the opening moves of patient diagnosis and management to involve some form of tension to pressure the candidates. A bank of eight CBDs was developed, each one following the same format. Each case opened with a paragraph describing an acute clinical situation. The candidate was able to read this outside the room in the three minutes prior to starting the station. During the CBD the candidate was presented with three further sections of information, one of which took the form of data, such as biochemical results, a chest X-ray or an electrocardiograph, and the others involved clinical progression of the case. The last section of information would produce a development of the case with some form of problem, such as an issue about clinical management or a clinical development that would create a conflict with previous issues.
The communication station was designed to observe interaction with a surrogate patient or relative in a clinically relevant scenario. The specialty training level 1 (ST1) candidates were given a simple communication task but with some elements of tension or anger in the scenario, such as discussion of the risks and benefits of steroid therapy with a patient's daughter who was averse to steroids. The ST2 candidates had cases that involved an ethical tension such as a discussion about implementation of gastrostomy feeding or a ‘do not resuscitate’ order. The candidates had three minutes to consider the case before starting the interview and, after nine minutes with the surrogate, there was three minutes' discussion of the case with the assessors.
There was a three-minute break between each 12-minute station. For the CBD and the communication station this time was utilised to start the candidate on the case. All assessors were asked to fill out a feedback form about their experience. Assessors were asked about the station they had undertaken and their assessment of the whole process.
Feedback results
In total, 69 consultant physicians took part in the interviews and 53 assessors returned the feedback forms. Some assessors gave feedback on more than one station. Not all people filled out the whole form and, in particular, people often failed to give a comparison with the traditional interview (16 out of a total of 68 returned forms). The reason for not filling out this part of the form was not elucidated and may simply have been neglect. The results that follow were from feedback forms provided during both rounds of the selection process.
All 53 who gave feedback were consultant physicians with a median of seven years from appointment (range 1–28 years). Twelve were from the large regional teaching hospital trust in Newcastle, the others from district general hospitals, of which one (James Cook University Hospital) is a large sub-regional hub. Thirteen were PACES examiners. All estimated their previous experience in recruitment: 21 had been involved many times in the past, nine on several occasions (over 10), 18 on a few occasions (less than 10), and five had no prior experience. The majority felt that the content of the assessment tools was useful and clear and over 90% felt that the three tools were each individually useful in assessing the candidates (Table 1).
Feedback results for individual stations.
Assessors were asked how this process compared to the previous traditional interview (Table 2). No comparison was stated on 24% of returned forms. When compared, the verdict was generally favourable: 57% of those who made a comparison felt that the structured interview was better than the traditional interview, 67% felt that the communication station was an improvement, and 84% felt that the CBD was an improvement. Overall, 69% felt that the new stations were an improvement on the traditional interview and 4% felt that they were worse. There was no correlation between those who felt that the process compared well with the traditional interview and those who had previous experience of PACES (an exam which many physicians are familiar with and which involves a similar process of assessment by scoring performance at separate stations). A selection of free-text comments is given in Table 3.
Comparison with traditional interview.
Free-text comment by assessors.
Discussion
In recent years interviews have been the sole mode of selection to general and higher specialty training in medicine in the UK. The growth of good practice in recruitment has eliminated those other sources of information about candidates which might have been considered in the past, such as personal knowledge of the candidate, third-party discussion and written references considered at, or before, the decision to appoint. Interview, used alone, has limited capacity to enable the candidate to demonstrate all the relevant skills and attributes needed for the post.4–9 Competency-based assessment has been utilised in the UK for selection in surgery, neurology, paediatrics and general practice and, in the last of these, validated against performance in post.8–11 The use of an assessment centre, incorporating a variety of assessment tools and including, usually, some form of interview, is now common practice for selection into general practice in the UK. The process is relatively new in relation to the selection of physician trainees. This study offers subjective feedback from a large selection process involving the impressions of over 50 physicians. Most of these had experience of undertaking the traditional interview for comparison and all had been through such interview processes themselves.
The striking finding is that a clear majority (over 90%) of assessors felt that the individual tools provided for the assessment of candidates were useful and effective and this is borne out by the free-text comments. The free text also identified that assessors felt that the process was objective and that the candidates were being given a fair chance to demonstrate their abilities. There was less unanimity when assessors were asked to compare the process to the traditional interview. Assessors missed the opportunity to be involved in all stages of the assessment of individual candidates and the freedom to explore a line of enquiry with individual candidates in the interview. It is of note that most dissatisfaction (in comparison with the traditional interview) was expressed when assessors compared the old and the new interview processes. Those parts of the assessment that were new and which, therefore, added to the process of assessment, were regarded more positively. Assessors also expressed some anxiety about not being able to gauge the total performance of a candidate and the lack of a mechanism to identify anyone deemed at one station to be absolutely unappointable – the system relied on cumulative poor scores meaning that such a candidate would not rank high enough to be appointed.
In the second wave of applications (round 2) there were attempts to tackle some of the criticisms levelled in the feedback from round 1. Some reflect intrinsic differences between the new and the old systems and the necessary rigidities of the assessment centre approach. However, there was concern that there was no opportunity to discuss individual candidates during the process of assessment: the system only recognised the individual scores taken in isolation. In round 2 the number of candidates in the day was reduced and time was built in to natural breaks for candidates to be discussed before starting a further tranche. However, this was found to be unworkable as there were so many scores to be entered for each candidate that it was impossible to get them collated in time to enable discussion at timely points. No attempt was made to reserve the discussion until the end of the day as it was felt that it would be impossible to expect assessors to be able to remember, and have meaningful discussions about, 24 or more individual candidates. More administrative support would be needed to enable scores to be available in real time and would work better with a simpler scoring system that required less data entry.
Another criticism raised was the feeling of loss of flexibility in being able to take the interview forward in any direction felt desirable at the time. This is partially the nature of a structured interview but is, in fact, more a reflection of the constraint of working to a rigid timetable. In mitigation, overrunning has bedevilled SHO interviews for a long time and many assessors commented informally on the efficiency of the assessment centre approach.
A further criticism was identified by the comment ‘bad luck for those who have not managed a similar case before’. However, after discussion, it was agreed that all the cases in the CBD were truly the sort of general acute medicine that the candidates should have been exposed to and that they had two other stations at which to perform well – in the past, there had been no such chance for redemption. It would be possible to replace one lengthy CBD with two shorter ones to cover more topics but the depth and complexity of the CBD was something that the assessors felt was part of its discriminatory power.
The assessment centre approach for selection of trainees in medicine described above has now been in place for over a year and some minor problems need to be solved. The number of scoring domains will be reduced and it should be possible to have meaningful and contemporaneous discussion of each candidate so that the system does not depend upon numerical scores alone. The use of a digital photograph of each candidate will facilitate this process. A more fundamental concern is whether these particular assessment stations are actually the best that can be devised for physician selection. In general practice multiple and complex forms of assessment are commonly undertaken including, for example, videoed simulated consultation with a patient, group exercises to assess group dynamic skills, and written exercises.11 These have been arrived at over many years. The process of deciding what would be the best assessment stations for physicians has barely begun and, regrettably, it will not be possible to undertake a direct comparison of new and old methods.
- © 2009 Royal College of Physicians
Jump to section
Related Articles
Cited By...
- No citing articles found.