The health problems associated with physical inactivity are well known and include ischaemic heart disease, obesity and diabetes.1 However, activity levels in England are low, with just over a third of men and a quarter of women achieving the government-recommended levels of physical activity.2 Those from the most disadvantaged backgrounds participate in the lowest levels of leisure time activity.1 3 Recognition of the economic consequences of the ill-health burden associated with inactivity has led to worldwide developments to improve health and reduce health inequalities by tackling sedentary lifestyles.4 In England, the government has launched numerous physical activity initiatives including exercise referral schemes, pedometer schemes and, most recently, health trainers5 (box 1).

Soon after coming into office, New Labour introduced a commitment7 that government policy must be evidence based, properly evaluated and based on best practice. Herein lies a problem: while there is good evidence of the health benefits of leading an active lifestyle,1 there is limited evidence for the effectiveness of physical activity initiatives in terms of their ability to improve health and reduce inequalities.8^–10 This gap between describing the problem and evaluating the effectiveness of proposed solutions is not unique to this public health issue. A recent review found that only 4% of public health research in the UK focused on interventions rather than simply documenting problems and that just 10% of the intervention research measured outcomes.11

The methodological difficulties and practical constraints of conducting good-quality evaluations of public health and social initiatives have been well documented.12 Another reason why so little high-quality evidence for the effectiveness of such initiatives exists is the over-riding influence of the political imperative.11 The failure of government to fulfil its own remit of evidence-based policy-making across the raft of NHS reforms introduced from the year 200013 and the rationale for the selective use of evidence in policy-making7 has been extensively discussed.14 Factors include the need to address pragmatic considerations such as cost and time constraints, as well as recognising the influences of pre-existing beliefs and underlying values, and political timeliness.15^–17 These constraints are not unique to England: in the United States and Canada, explicit attempts have been made to incorporate evidence into policy by establishing independent national bodies to provide summaries of the scientific evidence on health promotion.18 However, these have had variable impact because, for example, funding for evidence-based policies is not always forthcoming.18

While the limited impact of evidence on policy has been examined in depth, the impact of policy on the ability to gather evidence has attracted less attention. We demonstrate how the evolution of policy in public health can impede the evaluation of the effectiveness of public health interventions. The impact of a single key policy decision has been noted previously with respect to the “Sure Start” programme. The government chose to establish Sure Start as a universal area-based intervention for all young families living in designated areas.11 19 This was against the advice of research advisers who advocated random allocation of communities to the programme to allow systematic evaluation of outcomes.19 We present an in-depth case study of another high-profile public health initiative, exercise referral schemes (ERSs). Our analysis demonstrates how a series of policy decisions evolved in parallel with and with little reference to the development of evidence. We also demonstrate how this lack of convergence now precludes the gathering of meaningful evidence of effectiveness.

METHOD

We retrieved peer-reviewed articles, national policy documents, reports, press releases and guidance relating to ERS research, policy and practice development in England from 1994 to 2007 in order to establish an interpretative account of events. We adhered to a comprehensive literature search strategy to minimise the likelihood of selection bias. This comprised:

• Searches of electronic databases including Medline and Web of Science.

• Searches of websites for grey literature (eg Department of Health, National Institute for Health and Clinical Excellence).

• Retrieval and cross-checking of references cited in published articles and reports.

• Follow-up of citations recommended by researchers and experts in the field.

In addition:

• We used direct and referenced quotes to avoid the potential for inaccurate paraphrasing or de-contextualisation.

• We discussed our initial interpretations with experts in the field and, where necessary, clarified issues with authors of the peer-reviewed and grey literature we examined.

POLICY IMPEDING EVIDENCE: THE CHRONOLOGY OF EVENTS

Table 1 outlines the chronology of key research, policy and practice events concerning ERSs in England.

ERSs were introduced in England in the early 1990s and expanded rapidly within a few years.6 20 21 23 In 1994, Iliffe and colleagues21 published the first editorial on exercise referral to raise concern about the opportunity costs of funding an unevaluated public health initiative. However, rapid expansion continued unabated and was encouraged by a Department of Health (DH) press release in 2000 which expressed the government’s keenness to extend the number of schemes in operation.25 The press release paid lip service to the need to ensure effectiveness, but this stipulation was overshadowed by the headline message that exercise referral was a good thing, which should be expanded. In 2001, DH published the National Quality Assurance Framework for Exercise Referral (NQAF), which emphasised the importance of evaluation.4 26 However, the awareness, skills and expertise required to ensure that ERSs were able to build in evaluation mechanisms were rarely cultivated within local schemes,29 and the NQAF failed to achieve consistency and comparability of audit and evaluation mechanisms across the country.4 29 38 Researchers4 6 8 23 warned that the policy to promote ERSs was not underpinned by good evidence of effectiveness, but the proliferation of schemes continued.28 29 The prominence of ERSs was boosted further by their inclusion in the White Paper Choosing Health.5

The next White Paper, Our Health, Our Care, Our Say, published in 2006,31 continued the mantra of expansion and now, inexplicably, justified the policy in terms of the ”success„ (not defined) of existing schemes. These public assertions of “success” were published despite the fact that a DH co-funded evaluation of Local Exercise Action Pilots (LEAP) (which included five ERSs)34 and a NHS Research and Development Health Technology Assessment (HTA)-funded single centre randomised controlled trial (RCT) had not yet reported their findings. In 2005, DH had also commissioned the Public Health Interventions Advisory Committee arm of the National Institute for Health and Clinical Excellence (NICE) to survey the evidence and to make recommendations about the use of ERSs. NICE published their evidence review,30 which led them to conclude that there was insufficient evidence to recommend ERSs for the promotion of physical activity (unless they were part of a trial to determine effectiveness) in the same month that DH was trumpeting the success of the schemes in Our Health, Our Care, Our Say.31

The NICE guidance caused deep anxiety among schemes, and the implementation advice published by NICE in 2006 acknowledged their concerns.32 This resulted in confusing guidance. On the one hand, NICE recognised the contribution that schemes were making to multiagency working and warned commissioners to consider the implications for good partnership arrangements before withdrawing funding, while at the same time, they recommended that commissioners should only endorse “schemes to promote physical activity if they are part of a properly designed and controlled research study to determine effectiveness.”32 A year later, the DH Best Practice Guidance clarified the situation by stating that the requirement to be part of a controlled study applied only for those schemes existing solely for the purpose of promoting physical activity in people with no underlying condition or risk factors. All other schemes could continue as before, in accordance with the NQAF.37 As it is extremely unlikely that any schemes exist solely for healthy people, this statement from the DH allowed schemes to effectively ignore the (DH commissioned) NICE guidance.

IMPLICATIONS

Evaluation of the effectiveness of ERSs in England in terms of improvements in health and reductions in heath inequalities is now an unrealistic aim for several reasons.

First, they have been widely established across England. A recent national survey reported that 89% of primary care organisations in England had an exercise referral programme.28 Although the response rate to this survey was just 62%, our own survey of schemes across Greater London found that 30 out of 31 primary care trusts (PCTs) had an ERS running or in development.29 If this coverage is reflected across the country, then identifying areas that have not established schemes will be difficult. It could be argued that within PCTs there are general practitioners (GPs) who do not participate in the scheme because of clinical uncertainty about effectiveness or capacity constraints of existing programmes. Such GP practices may be willing to be randomised to a controlled evaluation of effectiveness. However, the extensive and well-publicised presence of these schemes could hamper recruitment to a randomised study. Contamination could also occur because patients who are not randomised to exercise referral by the GP can be referred via other routes (eg by physiotherapists).

Second, even if suitable sites for conducting a controlled evaluation of effectiveness were identified, funding for such a study is likely to be hard to come by. The DH has already contributed to the £2.6 million LEAP evaluation,34 and the HTA funded a single site RCT10 (referred to above). The HTA trial found that referral to an ERS added no additional benefit to simply providing advice on physical activity. The LEAP evaluation suffered from serious methodological difficulties which cast doubt on the value of its conclusions (table 1).34 35 Even so, the notion that more DH funding will be forthcoming is unlikely. In the past, the Medical Research Council (MRC) has funded applied public health research; however, the division of responsibilities between the National Institute for Health Research (NIHR) and the MRC implemented as a result of the Cooksey review39 means that, from now on, such research will be funded by the NIHR only. NIHR is the new health research agency of the DH. PCTs and local authorities are unlikely to fund evaluations of their schemes in the face of funding shortfalls and the diversion of money earmarked for lifestyle interventions to acute services.40

Third, even if sites for evaluation were identified and research funding was found, the results are unlikely to have an impact on the provision of existing schemes. Dismantling schemes would have profound effects that go beyond simply saving the costs of an ineffective intervention because hard won and effective partnerships between PCTs and local authorities would be damaged.29 32 Modifying existing schemes would also be difficult, akin to “unravelling and reknitting a cardigan while we continue to wear it”, as described by Muir Gray with respect to the similar situation facing some screening programmes (Smith,41 p. 358).

Fourth, analysis of referral to and use of exercise referral by disadvantaged groups is required to examine the effectiveness of targeted access, a frequently cited objective of local ERSs to improve health inequalities. The NQAF also stated that equity should be a primary consideration in service design and delivery.26 However, a comprehensive analysis of equity across schemes is not possible because guidance on the collection of sociodemographic data using standard definitions has never been issued. The result is that data collection is incomplete and inconsistent across schemes.29 38 For the 30 PCTs in Greater London that run an ERS, only eight collect information that allows assessment of access and use by different social groups.29 The remaining schemes do not collect either sociodemographic data or data on uptake after referral or on adherence.

If the experiences of ERSs are ignored, then mistakes could be made again with other public health interventions. There are already signs that this could happen with another initiative. The Health Trainer Scheme was launched in the 2005 White Paper Choosing Health5 (box 1). Initially targeted at the most disadvantaged areas, in 2007, the DH allocated funding to all PCTs to allow them to establish schemes despite the fact that the results of a DH-commissioned preliminary audit of activity in this area was not available until November 2007 and a synthesis of local evaluations not available until spring 2008. The DH is planning to fund a comprehensive evaluation of the clinical and cost-effectiveness of health trainers but, once again, its commitment to universal provision precludes the ability to undertake an evaluation using a randomised controlled design.

CONVERGING PARALLEL LINES

On the one hand, the DH has shown a continued commitment to accumulating and appraising research evidence by funding a succession of relevant initiatives. In parallel to this stream of enquiry, the DH has promoted the expansion of an intervention without sufficient reference to the evidence it commissioned. In doing so, the ability to accrue meaningful evidence to support policy-making has been undermined. This situation has probably occurred for the well-known reasons outlined in the policy literature.15 17 These include the pressure to achieve outcomes within short timescales, to satisfy the public and health professionals, to be seen to take action and to appease powerful lobbies.15 In the context of ERS, policy-makers were faced with the unenviable situation of having insufficient evidence of the effectiveness of interventions to tackle sedentary lifestyles, coupled with an imperative to be seen to be taking action to tackle major and increasing problems such as obesity. It is therefore not surprising that some policy analysts are resigned to the inevitability of the illusory nature of evidence-based policy-making.42 However, other policy-makers have shown that it is feasible to bring the parallel tracks of evidence and policy-making into closer convergence.43 44

For this to occur, a shift towards an “evaluation culture” is required.16 This is not an unrealistic recommendation. The World Bank now stipulates that it will only fund projects that have appropriate evaluation and monitoring built in.44 A range and combination of quantitative and qualitative research methodologies are likely to required.12 Where randomisation is not possible or appropriate, the next best research method should be used. But it should not be assumed that randomisation is never possible on the basis that the public will not accept being denied a plausible intervention on a random basis while it is being evaluated among their neighbours. Important lessons about working effectively with communities can be learnt from the Welsh Assembly Government’s Primary School Free Breakfast Initiative.43 Rather than being implemented in all schools immediately, this is taking a pilot approach including a cluster randomised trial of the intervention.45

With respect to exercise referral, the Welsh Assembly Government has also funded a national development and evaluation of ERSs, which employs a randomised design and includes the requirement for all included ERSs to adhere to national guidance on data collection. This convergence of implementation with evaluation allows the development of nationally agreed standardised minimum datasets containing sociodemographic data, process and even validated outcome measures. These can be established at the outset and enable high-quality routine monitoring and evaluation of schemes in routine service delivery.

If we are to avoid perpetuating the current situation in which political decisions prevent the accumulation of evidence about effectiveness, researchers need to become adept at “managing the political terrain”.46 This is a resource-intensive and long-term task that researchers cannot do alone. To be effective, collaborations are required with established and successful advocacy groups (such as the National Heart Forum) and with stakeholders. A political analysis of stakeholder interests is therefore a key primary task. Working together, stakeholders can ensure that public awareness is maintained and that systematic approaches are employed in getting and keeping the evaluation issue on the policy agenda.

Plausible and well-meaning public health interventions may not work; they could produce harmful effects or widen health inequalities. The political argument for controlled roll-outs of public health interventions must now be won.