ABSTRACT
The General Medical Council (GMC) national trainee survey (NTS) monitors junior doctor training experience annually, which is then used by organisations such as Health Education England to inform quality management. Its validity as an assessment of the learning environment to drive improvement is frequently questioned; currently there are no published evidence-based studies to demonstrate its impact. To explore the effects of the GMC survey, we carried out a retrospective cohort study using publicly available GMC NTS survey data. We compared 2018 and 2019 scores in paediatrics in London across all 18 survey indicators, to identify any relationship between these 2 consecutive years of data. Our findings demonstrate that results of the GMC NTS in 1 year are associated with a change in the NTS the following year, with both an improvement in below average departments and deterioration in above average units. These findings suggest that annual GMC NTS results may have an impact on the quality of learning environments as measured in subsequent surveys – therefore they act as both a measure and a potential modifier of outcome.
Background
The General Medical Council (GMC) national trainee survey (NTS) monitors training experience annually.1–3 Using an online platform, all trainees are asked to rate the training conditions at the site at which they are working using rating scales. The survey is sent to all 75,000 doctors in training in the UK, and has a response rate of 94.8%.5 The responses are grouped into 18 indicators, including supervision, rota design, and curriculum coverage. The results are rated using standard deviation calculations into red, pink, white, moss and green (Fig 1).6
Explanation of the General Medical Council national training survey colour scheme.1
Survey results are used, in conjunction with other tools, by Health Education England (HEE) and the education and training governing bodies in the devolved nations, to explore the quality of postgraduate training, sometimes leading to changes in where trainees work and are trained.7 Nevertheless, the validity of the data is questioned with issues raised around, variously ‘forcing’ respondents to answer; the survey not being taken seriously by trainees; maintaining confidentiality; sample size; and question wording.2,4
There is anecdotal evidence that positive scores may encourage trainers in the department to maintain and improve standards of training; there is also a perception that ‘white’ scores, in particular, may lead to complacency.8 Similarly, poor results may demoralise departments under significant pressure, particularly with staffing issues, but can also be a stimulus for change.9 The survey results may ‘fall on deaf ears’; amid a busy, fraught department with competing priorities and an already stretched workforce, junior doctors and their training will often not be prioritised.10 We wanted to explore how NTS results in 1 year might relate to changes in scores the following year.
Objectives
We aimed to use GMC survey data in 2 consecutive years to examine whether GMC results may be associated with influencing a change in performance within a training department.
We had a null hypothesis; GMC NTS results are not associated with bringing about change in educational and training performance of a clinical department.
Methodology
We downloaded raw scores for all 18 indicators in the NTS in 2018 and 2019 for paediatrics from the GMC website.1 We chose to look at trainee feedback (specialty trainee years 1–7 (ST1–7) as a run-through programme) at all levels, received from the paediatric specialty in all hospital sites within London. The score for each site was compared with the specialty's national mean. We calculated the standard deviation (SD) for each score at all 33 hospital sites in London and then the Z score for each indicator at each site, as a measure of that indicator's deviation away from the national mean (eg Z=0 signified that a value is equal to the national mean, Z=1 indicated that a value is one SD above the national mean, and Z=−1 was one SD below). Using the Z scores in 2018 for each indicator as a baseline, we calculated the change in Z scores in 2019 by indicator by site (Z score in 2019 minus Z score in 2018).
Where an indicator had fewer than three respondents replying, no score was available so all raw scores for the indicator for both years in that respondent group were excluded from analysis. A total of 606 Z scores could have been calculated for each year (total number of scores was 1,212) of which 11 had fewer than three respondents. This resulted in a total of 595 Z scores per year, therefore 595 data points were used to calculate the Pearson's r value shown in Fig 2.
Relationship between Z score in 2018 (x-axis) and change in Z score between 2018 and 2019. Pearson's r = −0.60.
Results
The data showed that departments with survey indicators below the national mean were more likely to improve in the following year than other departments; there was a significant association between the distance of indicator score from the mean and the observed improvement (Fig 2). Conversely, departments with indicators above the mean tended to show deterioration in their survey results in the following year. Pearson's r value was −0.60 indicating a moderately strong inverse correlation between a poor Z score in 1 year and an improved Z score the following year. The relationship between higher baseline scores and subsequent deterioration was weaker.
The improvement in poorly performing departments was more marked than the deterioration of units who performed well, therefore the data were re-plotted, separating out indicators that were above and below the mean in 2018 (Figs 3 and 4, respectively). This demonstrated that indicators with positive scores in the GMC NTS in 2018 are only weakly associated with a worsening of performance the following year. This analysis also showed that indicators with scores below the mean in 2018 showed a more substantial improvement (Pearson's r value of −0.53 compared with −0.35) for the whole cohort.
Relationship between national training survey scores in 2018 and scores in 2019 for trusts with above average national training survey scores. Pearson's r = −0.35.
Relationship between wwww training survey scores in 2018 and score in 2019 for trusts with below average national training survey scores. Pearson's r = −0.53.
Discussion
The findings of this study demonstrate that the results of the GMC NTS in 1 year were associated with change in the survey results the following year in a single consecutive year analysis, with evidence of improvement in below average departments and deterioration in above average units. This study appears to suggest that GMC NTS results may influence the performance of a department as measured in the subsequent survey year. There are however a number of possible reasons for these findings. One explanation is that these results may reflect the response of an organisation to the survey results, using the dataset as a stimulus for change in a department that may be struggling to maintain an effective learning environment and, for others, may reflect relative complacency in departments that perform well, or within the ‘normal range’. Departments that performed well in the first year showed a weaker correlation to a change in results in the subsequent years; this may be due to more sustainable mechanisms in place to correct issues and ensure continued good performance. Another cause for change in results is a cohort effect ie every year there will be changes in the group of trainees as they rotate placement, enter or leave the specialty.
There are a number of limitations to this study. Firstly, there may be errors in survey completion, leading to erroneous values in one group and then ‘regression to the mean’ or normalisation of scores the following year – although the different results seen in analysis of below and above average indicators makes this unlikely. Secondly, the study looked at only one year-to-year comparison, in one specialty in a single region; it is possible that analysing data across several years of survey data may generate different results. Thirdly, this finding may not be applicable to other training programmes, particularly internal medicine and foundation training, in which trainees move through a number of different departments, sites and organisations each year with varying levels of specific interest in the post they are placed in at the time of the survey. The NTS asks trainees to respond to the questions based on the current department in which they work, and in many – particularly true in paediatrics – the trainees will have rotated into new posts only a few weeks before the survey opens in March; for this reason, many schools of paediatrics have run their own surveys at a different time of year. Further analysis of these school surveys and comparison to the GMC data would allow for the findings of this study to be interrogated in more detail.
Other future work could also focus on organisation level data, as approach to below average survey results is likely to vary by factors within trusts and hospitals, such as staffing levels and skill mix, educational leadership, funding and service pressures. The statutory education bodies (Health Education England, Health Education and Improvement Wales, NHS Education for Scotland, and Northern Ireland Medical and Dental Training Agency) each review GMC survey data annually and implement quality review and improvement programmes based on the findings of the survey. Each of the four nations have individual approaches, and there are also regional variations within each of these arm's-length bodies. Further analysis to look at potential geographical differences would therefore be of value when considering the variation in interventions.
Conclusion
Results in the GMC NTS in 1 year may influence the likelihood, direction and degree of change in the following year, particularly in departments with comparatively poor performance, which are significantly likely to show improvement in subsequent years. It may also be that errors within the survey lead to aberrant values, which then lead to a natural normalisation of scores the following year, although this would not explain the different effect (up and down) as well as the difference in magnitude of the effect (Pearson's r value of −0.35 and −0.53) on below and above average indicators. An effective survey is not defined just by the quality of data and analysis, but also how it can impact improvement for the population being surveyed. This study provides evidence for such an effect in the GMC NTS in London paediatric departments, and is worthy of investigation in other training programmes and regions.
- © Royal College of Physicians 2020. All rights reserved.
Article Tools
Jump to section
Related Articles
Cited By...
- No citing articles found.