Abstract
Peer review has been widely employed within the NHS to facilitate health quality improvement but has not been rigorously evaluated. This article reports the largest randomised trial of peer review ever conducted in the UK. The peer review intervention was a reciprocal supportive exercise that included clinicians, hospital management, commissioners and patients which focused on the quality of the provision of four specific evidence-based aspects of chronic obstructive pulmonary disease care. Follow up at 12 months demonstrated few quantitative differences in the number or quality of services offered in the two groups. Qualitative data in contrast suggested many benefits of peer review in most but not all intervention units and some control teams. Findings suggest peer review in this format is a positive experience for most participants but is ineffective in some situations. Its longer term benefits and cost effectiveness require further study. The generic findings of this study have potential implications for the application of peer review throughout the NHS.
Background
Peer review has been employed both within the NHS and overseas as a mechanism for affecting health quality improvement but there is surprisingly little evidence for its effectiveness. Previous reports have detailed process or the perceptions of those involved as indicators of success.1,2 Evidence relating to the success or limitations of peer review in effecting service improvement may have significant implications for the NHS quality assurance programme. The National Chronic Obstructive Pulmonary Disease (COPD) Resources and Outcomes Project (NCROP) is a tripartite initiative from the Royal College of Physicians (RCP), British Thoracic Society (BTS) and British Lung Foundation (BLF) that aimed to evaluate if targeted mutual peer review brings about faster change in service development than the usual mechanisms that operate within the NHS.
Methods
Details of the NCROP study design and methods are described elsewhere.3 The project was overseen by a steering group of COPD professionals and managers including Department of Health representation. Briefly, all UK hospitals admitting acute COPD cases were invited to participate in a prospective randomised trial of focused mutual peer review. Peer review teams consisted of a lead respiratory consultant, another healthcare respiratory professional (usually a nurse or physiotherapist), a hospital service manager, a primary care commissioning representative, and a patient representative. The group identified four key COPD service areas from previous UK national COPD audits highlighted as having variable provision across the country to be the focus of peer review. Each has a good evidence base and national guideline recommendations.4–13 These services were non-invasive ventilation (NIV) for acute type 2 respiratory failure treated in hospital, facilitated (early) discharge schemes (EDS), pulmonary rehabilitation (PR) across both secondary and primary care interfaces, and assessment for out-of-hospital oxygen requirement. Quality indicators were identified for each, drawn mainly from national guidelines with others agreed by consensus, a total of 46 in all. Another area of interest, palliative care for COPD, was added at the request of BLF patient groups, to record current practice and planned improvements but without indicators of quality.
Reciprocal review visits were arranged within a four-week period. A semi-structured programme for the visits was developed by the steering group. This directed discussion towards the COPD service areas but also allowed for wider discourse about the COPD service in general. In advance, visiting teams received a unit's self-completed baseline proforma during spring 2007 describing their service provision and attainment (met in full, partially met, not met at all) of the quality indicators. A joint meeting at the end of each visit produced recommendations and a service development plan supported by all parties was then required within four weeks.
Units first submitted service provision data to NCROP in November 2005. The number of service areas provided by units (of NIV, PR and EDS) ranged from zero to three and to maximise the potential for units to learn from each other those with zero/one service were paired to those with all three, units with two services were paired to units with all three and then all remaining units were paired. Where possible, paired units were located within 2.5 hours travel of each other, but not in adjacent districts. Each pair of units was randomised 3:2 to either the reciprocal peer review group or to the control group.
Project outcomes comprised changes in services offered and changes in quality indicator scores, between baseline results collected during the peer review visits from spring 2007 and national COPD audit results from spring 2008 (www.rcplondon.ac.uk/clinical-standards/ceeu/Current-work/ncrop/Pages/audit.aspx). All trial units were asked to complete a change diary 12 months after the peer review visits programme had finished, to provide information about major service changes occurring during the year, any influence of NCROP on developments, and any other impact of NCROP involvement on the COPD team.
Quantitative data were analysed using SPSS v.15. Qualitative data were processed using grouped themed independent analyses by two (CMR, JMP) of the researchers.
Results
In total, 106 hospital units volunteered to participate. Six withdrew before randomisation. Units consisted of either a single acute hospital site or more than one hospital within an acute trust. Fifty-four units (27 pairs) entered the intervention arm and 46 (23 pairs) the control arm. Fifty-two peer review visits took place within the designated 12-week time frame and only two were delayed. All 54 visits included a respiratory consultant lead, 51 a respiratory nurse or physiotherapist specialist, 39 a hospital service manager and 30 a primary care organisation commissioning representative. Patients were represented at 20 visits. All teams finalised an agreed service development plan within four weeks of the visit.
The 2007 baseline results (not shown) indicated that randomisation produced similar groups of hospitals in terms of fully meeting specific quality indicators and in quality indicator scores for three service areas but that pulmonary rehabilitation services were better provided for in the control group (p=0.02, Mann-Whitney U test).
There were no statistically significant differences between the two groups for the change in service area quality indicator scores between the 2007 baseline NCROP survey and the 2008 national audit (Table 1). In regard to individual standards there were mainly minor differences between intervention and control units. There were better changes reported for 30 indicators in favour of the intervention group (4 NIV, 8 PR, 7 EDS, 11 long-term oxygen therapy (LTOT)) and for 12 indicators in favour of the control group (5 NIV, 3 PR, 2 EDS, 2 LTOT) with four sets favouring neither group (3 NIV, 1 LTOT). For six indicators differences between intervention and control units were of statistical significance (0.01<p<0.05) or of borderline significance (0.05<p<0.10) and all favouring the intervention group.
Hospital variation in chronic obstructive pulmonary disease quality indicator scores for intervention and control units.
Unit diaries were returned completed and analysed for 43 of 46 control units and 50 of 54 intervention units. Data were analysed under the two headings within the diaries: changes during the one-year period post-peer review and impact of NCROP peer review on changes and results (Table 2).
Sites reporting significant service changes within the year following peer review visits in control and intervention units and the settings in which changes occurred.
Service changes perceived as negative were reported in 5/50 (10%) of the intervention units and in 7/43 (16%) of the control units. Negative changes were grouped as:
staffing: usually long-term vacancies, sickness or cuts eg ‘systematic loss of nursing posts’
relationships: often managers and clinicians or primary and secondary care ‘failure to maintain regular management meetings’
reconfiguration: both within the hospital unit and at the primary care trust ‘the medical service has experienced major disruption as a result of reconfiguration…’, ‘It feels as if we are swimming in molasses. Nothing seems to be moving anywhere’.
Positive service change was reported in 37/50 (74%) of the intervention sites and 13/43 (30%) of the controls. Examples of how NCROP had helped bring about positive changes in service and within departments were grouped into themes described in Table 3.
Examples of how peer review has helped bring about positive change in service and within departments.
Discussion
NCROP represents the largest ever voluntary peer review study run in the UK. It was completed within a short time frame and with positive engagement of the vast majority of clinicians. Managers from acute trusts attended most peer reviews and primary care organisations (PCOs) were represented in most but not all visits perhaps reflecting the restructuring of PCOs that occurred during the project. It has been more difficult to recruit and actively involve patients in the process and more thought is required as to how best to utilise their expertise in service improvement programmes of this kind.
The outcomes of the peer review process on the individual participants and on service change is mixed. The quantitative analysis of the rate of change in service development shows little difference between the intervention and control groups. There are several possible reasons: peer review may have failed to influence service, it may be that services within the NHS are in continuous dynamic change, some for the better, some for the worse and a randomised controlled trial may not be powered sufficiently to demonstrate benefit in this state, or it may be that more time is required for significant service change to come about.
The qualitative evaluation of the change diaries provides evidence to support each of these hypotheses. They reveal the variation in the rate of change within respiratory services with some services struggling to progress and going backwards – ‘…swimming in molasses’ – while others are clearly progressing rapidly with healthy interaction between clinicians, hospital management and commissioners (Table 2). This dramatic variation in the rate of change poses a challenge to the quantitative evaluation of a change methodology such as peer review.
Some sites saw no benefit from peer review because they felt they already had service development well underway or because they felt there was no way they could make progress as their plans were falling on deaf ears.
These negative responses were greatly outweighed by the sites expressing a positive view and reporting many changes that were felt to be related to the peer review process (Table 3). Peer review was associated with more change in hospital-based services (ie NIV and palliative care) than community-based services (ie provision of LTOT and PR) (Table 2). The findings suggest that in many places the added value of peer review was predominantly on the hospital component of service delivery.
The pace of change within the NHS has previously been documented as requiring more than one year to see service developments.14 The change diaries indicated that developments were at differing stages of completion including: further discussions, submission of business cases, cases being accepted and change having taken place. It is quite possible therefore that the quantitative evaluation at one year was too early to detect differences in completed changes in service provision.
What is clear is that most participants in the peer review group and interestingly a sizeable minority in the control group found benefit within NCROP. The former for a wide range of reasons that include: the provision of a quality framework, sharing of good practice, the bringing together of commissioners and providers, team-focused outcomes of better morale and self-awareness of service provision. For the control group a quality framework and the stimulus to review services and benchmark against agreed standards were appreciated in some but not all sites. This suggests that simply providing tools such as a quality framework may not be sufficient to engage clinicians and a proactive dissemination may be required. The examples of a national strategic framework launched in Northern Ireland suggest this may well be the case and the forthcoming English strategy document is awaited with interest.15 A number of negative issues were also uncovered within NCROP that included a frustration with negotiations between the acute trust and the PCO and in some instances the internal dislocation of a vision between managers and clinicians. In the context of such poor relationships any service improvement initiative is unlikely to succeed.
Peer review on this scale is also not without its costs in terms of participant time and preparation and this report provides evidence on which its early effectiveness may be judged. It may be that it is too soon to measure the medium to long-term benefits of participation in NCROP or to appreciate if this process represents ‘value for money’. It is apparent that there are a number of aspects of notable practice both for clinical teams and for those responsible for commissioning and providing services that can be identified as facilitating or inhibiting service change. It is also apparent that service change often occurs over prolonged periods of time and clinicians involved in the process must commit to the long-term achievement of goals.
The NCROP peer review study was carried out in the clinical context of respiratory disease. However, the findings with regard to improvements to services and barriers to change, for example impact on team working, shared learning, objective data on which to base proposals for change, interaction between clinicians, managers and commissioners, are generic to all clinical conditions.
What emerges from this study of peer review is that, in the face of the highly volatile state of service change within the NHS, quantitative analysis at one year does not provide the whole picture. The qualitative analysis indicates that peer review as carried out within NCROP is associated with many in-hospital changes and is felt by participants to have many benefits, both in driving change and on the department. The challenge is to define which services are in a state to benefit from peer review to ensure that the intervention is focused where it can be most cost effective.
Acknowledgements
We are grateful to the NCROP steering group, the COPD teams and their patients who participated in this study and to the Health Foundation which funded the project.
- © 2010 Royal College of Physicians
Article Tools
