Changing PACES: developments to the examination in 2009
Abstract
As a consequence of change in medical, educational and regulatory practice, MRCP(UK) successfully modified the international PACES examination in 2009. This brief paper explains the rationale for change and summarises the development and implementation process.
Background
The PACES (practical assessment of clinical examination skills) examination was introduced in 2001 to provide a valid, reliable and deliverable assessment of physical examination and communication skills for trainees in general medicine.1 Specific advantages of the PACES format over previous forms of the MRCP(UK) clinical assessment were the standardisation of candidate experience, direct observation of the candidate–patient interaction throughout the assessment, and extension of the assessment of communication.
The examination continues to have a key role in the summative assessment of trainees in medicine around the world and forms one of the three parts of the MRCP(UK) diploma. In the United Kingdom, MRCP(UK) forms one part of the mandatory assessment system for core medical training (CMT), the other being workplace-based assessment.2 PACES is mapped to the curriculum for CMT3 in the UK and assesses clinical skills, applied clinical knowledge, and the attitudes and behaviours needed for the effective practice of bedside medicine. It builds on, and is complementary to, the two other, knowledge-based, components of the MRCP(UK) diploma.4
Twenty-five diets of the examination were held between 2001 and 2009, with over 35,000 candidates sitting the examination in the UK and in nine international centres. The mean pass rate for all candidates over this period was 44.4%. This paper describes the rationale for, and summarises the process leading to the implementation of, changes to the examination introduced in 2009.
Drivers to change
Internal review
In 2005, an internal review of the examination concluded that although the basic format was extremely successful, there were several areas where improvements could be made.
For a number of academic and practical reasons, it was felt that Station 5 was the least satisfactory of the five stations, and the review recommended its redesign. In addition, the pass standard had been based on a marking system that permitted candidates to compensate for a poor performance in a key clinical skill, such as communication, with better performance in others, for example physical examination. This was felt to be inherently unsatisfactory and, although only a small proportion of candidates passed on account of this, the review recommended the development of a marking methodology and pass standard in which such compensation could not occur.
The review also highlighted that at each PACES station, communication between the candidate and the patient was dissociated from physical examination. That is, the candidate either spoke to the patient but did not examine them, or vice versa. It was felt that the validity of the assessment would be enhanced if at least one station integrated the assessment of these two domains. Finally, the review suggested the need to develop a station at which content areas not represented in the original PACES format, such as acute or elderly medicine, could be represented.
External context
Three external factors further influenced thinking regarding the development of the PACES clinical examination, namely:
A new UK regulatory authority, the Postgraduate Medical Education and Training Board, established in 2005, introduced a series of principles, later to become standards, for postgraduate assessments.5 These standards were mandatory and included the requirement for all postgraduate assessments to have transparent, fair and evidence-based pass standards.
Competency-based training and assessment, in which performance is broken down into a series of discrete competencies, each of which can be individually taught, learnt and assessed, gained currency in educational circles.
Curricula for core medical and foundation training changed in their scope, content, focus and structure, reflecting, in particular, the emergence of the competency-based model described above.6–8 These curricula placed clear emphasis on the attainment of multiple distinct competences, each of which should be evaluated and all of which must be attained before progression to higher specialty training. In association with the introduction of these curricula, a number of new workplace-based assessments were introduced.
The changes to PACES proposed as a result of these considerations reflected the desire to design an assessment with content clearly mapped to the new curricula, that could meet new regulatory standards and which utilised an assessment methodology in which the pass standard was defined by attainment of a satisfactory standard in all skills (competences) assessed. The MRCP(UK) Clinical Examining Board (CEB) accepted the recommendations of the review group in 2006, that is, that a non-compensatory pass standard should be developed and that Station 5 should be revised.
Process of change
A sub-group of the CEB, the new PACES Implementation Group (nPIG), was tasked to lead the development and change process. This process included involvement of, and communication with, key stakeholders, interaction with the regulator, running of pilot examinations, development of new training materials for examiners and, for the first time, candidates.
The changes
Full details of the format of the examination can be found on the MRCP(UK) website.4
Station 5
Station 5 of PACES was reformatted to permit, for the first time, concurrent assessment of communication and physical examination skills, more accurately replicating real life tasks of trainees and further enhancing the validity of the assessment. In addition, the system content of the encounters at this station was no longer restricted, thus increasing capacity to sample broadly from the curriculum and assess candidates' clinical skills in a wider range of specialty specific, and multispecialty, contexts.
Two 10-minute encounters, each known as a brief clinical consultation (BCC), replaced four of five minutes, and each focused on a specific clinical issue arising in the care of a patient in an inpatient or outpatient setting. Interaction between the examiner and candidate was limited to two minutes, placing the emphasis of assessment on observation of the patient–candidate interaction, rather than questioning of the candidate by an examiner. In addition, the inclusion of an element of communication in these encounters further extended that overall assessment of communication in the examination.
Assessment methodology
In the original PACES format, examiners awarded separate marks for a variety of sub-domains of performance at each encounter, but these marks did not contribute directly to the candidate's final score. These sub-domains of performance were mapped to the anchor statements, or key competencies, underpinning the assessment. In addition to awarding marks for each of these sub-domains, examiners constructed an overall judgement mark based on the candidate's global performance at each encounter. Importantly, it was only this mark that contributed to the actual score and pass/fail outcome. In the new methodology, the performance sub-domains and anchor statement competencies were integrated and redefined as seven ‘core clinical skills’.9 The marks awarded for these skills now contributed directly to the pass-fail outcome and examiners were no longer required to formulate their judgements about performance into an overall judgement mark.
The marking scale was changed from a four-point scale to one of three points. In accordance with the concept of competence, examiners were required to award either a satisfactory (2 marks) or unsatisfactory (0 marks) judgement for each skill, with the option of awarding borderline (1 mark) to a candidate who attained some, but not all, of the criteria agreed necessary for the award of a satisfactory judgement.
As a result of these changes the number of individual examiner judgements contributing directly to the candidate's total score increased from 14 to 86, and the possible range of total assessment scores changed from 14–56 to 0–172.
In addition, it was possible, for the first time, to derive a total score for performance in each of the seven core skills thus permitting the creation of a pass standard based on performance in each skill (‘uncompensated’) rather than on total assessment score (‘compensated’). Figure 1 illustrates both methods.
Introducing the new pass standard
Transitional phase. Pilots of the examination had enabled successful evaluation of the new mark sheets and structure of Station 5 and provided data on candidate performance in each of the seven skills. An initial standard setting meeting, utilising a modified Angoff procedure and informed by results from the pilot examinations, derived a new pass standard based on total test score, and also seven individual pass standards for each skill. However, the limited numbers of candidates in the pilots did not provide sufficient data on which to base definitive new pass standards for each of the seven skills.
Therefore, in order to ensure a smooth transition to the new uncompensated pass standard, it was recommended that a transitional phase occur. This provided a mechanism to ensure that pass rate variation due to changes in examiner behaviour or assessment methodology could be monitored and minimised, and permitted collection of large amounts of data on the performance of candidates in the seven core skills in a real examination setting that could subsequently inform the refinement of pass standards for each of these skills.
For this reason, and with the approval of the regulator,10 a transitional one-year period occurred, in which the new station and marking system was introduced but the pass standard required remained one based on a compensated total assessment score, closely linked to the previous standard of 41/56.
The first three (transitional phase) diets. Over 4,500 candidates sat the new format of the assessment in over 100 centres in the UK and in nine international centres between October 2009 and July 2010.
The overall pass rate over the first three diets, utilising a compensated pass mark of 138/172 (80.2%), as defined from the pilots and first standard setting meeting, was well within the limits of historical annual variation at 42.6%. Examiner acceptance and understanding of the changes has been good, and has increased as familiarity with the details of the changes and opportunities to participate in the examination have increased.
Candidates have utilised the online preparation video,11 produced for the first time in association with the introduction of the new format of the examination, to a high level, with 60% of candidates around the world viewing the video before sitting. In exit surveys, 17% of candidates thought the examination would be easier, 4% harder and 79% about the same degree of difficulty. Concordance between examiners has remained very good, despite the increase in the number of independent judgements made, increasing to 77% overall.
As described above, the new marking methodology permits scores for performance in each skill and each encounter to be determined. The mean scores for the 4,615 candidates in the first three diets are shown in Table 1.
PACES pass standard after the transitional phase. In July 2010, with the results of over 4,000 candidates available, a further standard setting meeting took place in which senior examiners reviewed candidate performance, and the results of the initial Angoff procedure, and defined pass standards for each of the seven skills.
As a candidate could theoretically attain the seven separate pass standards yet attain a very low total score, an additional requirement to attain a total test score of 130/172 has currently been retained. The final details of the pass standard, and the means by which the pass standard is monitored and modified over time, are the subject of a further submission to the regulator in 2011.
Conclusion
MRCP(UK) have successfully modified the PACES examination to permit individual evaluation of a series of key clinical skills, assessed in a variety of clinical contexts in a summative assessment setting. The prime aim of this change is to ensure that passing candidates are competent across the range of clinical skills assessed. In doing so, integrated assessment of communication and physical examination skills has been introduced into one station.
All assessments must evolve as developments in training requirements, educational thinking and curricula emerge. In order that assessments change apace with such developments, bodies responsible for postgraduate assessment must carefully consider the processes and resources necessary to ensure changes are adequately piloted and evaluated and that examiners are adequately trained and candidates and other stakeholders are adequately informed. This paper summarises a period of five years of development and demonstrates the intensity of input and time necessary to make such changes in a manner that results in an assessment that is fair, reliable, valid and robust, and can meet the requirements of the current regulatory environment.
- © 2011 Royal College of Physicians
References
- ↵
- Dacre J,
- Besser M,
- White P
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Elder AT,
- McManus IC,
- McAlpine L,
- Dacre JE
- ↵
- Elder AT,
- McManus IC,
- McAlpine L,
- et al
- ↵
Article Tools
Citation Manager Formats
Jump to section
Related Articles
- No related articles found.