Article Text
PDF
Abstract
Objectives This paper reports on findings from a systematic review designed to investigate the state of systems science research in public health. The objectives were to: (1) explore how systems methodologies are being applied within public health and (2) identify fruitful areas of activity.
Design A systematic review was conducted from existing literature that draws on or uses systems science (in its various forms) and relates to key public health areas of action and concern, including tobacco, alcohol, obesity and the social determinants of health.
Data analysis 117 articles were included in the review. An inductive qualitative content analysis was used for data extraction. The following were systematically extracted from the articles: approach, methodology, transparency, strengths and weaknesses. These were then organised according to theme (ie, commonalities between studies within each category), in order to provide an overview of the state of the field as a whole. The assessment of data quality was intrinsic to the goals of the review itself, and therefore, was carried out as part of the analysis.
Results 4 categories of research were identified from the review, ranging from editorial and commentary pieces to complex system dynamic modelling. Our analysis of each of these categories of research highlighted areas of potential for systems science to strengthen public health efforts, while also revealing a number of limitations in the dynamic systems modelling being carried out in public health.
Conclusions There is a great deal of interest in how the application of systems concepts and approach might aid public health. Our analysis suggests that soft systems modelling techniques are likely to be the most useful addition to public health, and align well with current debate around knowledge transfer and policy. However, the full range of systems methodologies is yet to be engaged with by public health researchers.
- systems thinking
- complexity
- systems science
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This study provides a systematic review of the application of systems science and systems thinking to the field of public health.
The review identified critical shortcomings in the use of systems methodologies being used.
The review showed that public health is currently not engaging with the full range of systems methodologies.
The sample of articles is representative but not comprehensive, which is a limitation of the study.
Introduction
In the past 5–10 years, there has been rapidly growing interest in the applicability of ‘systems science’ to public health. Systems science is a broad class of analytical approaches that aim to uncover the behaviour of complex systems.1 ,2 A general distinction is made between ‘hard’ systems methodologies which refer to quantitative dynamic model building (ie, system dynamics) and ‘soft’ systems methodologies which refer to qualitative, action-based research methodologies (ie, critical systems heuristics). As a whole, systems methodologies are thought to enable researchers and decision makers to examine system components, and the dynamic relationships between them, at multiple levels, from cell to society.1
Taking a systems approach encourages a rethinking of organisations and system issues, including how actors behave in relation to them and are involved in their diagnosis and treatment.2–5 Here, the emphasis is placed on understanding the ‘whole’ system, rather than focusing exclusively on individual components.2–4 ,6–8
While a range of systems methodologies exist (or are in the process of being developed and refined within a public health context), the most advanced systems methodologies seek to model systems and/or subsystems, in order to identify potential points for intervention and change.i Modelling and simulations are thought to have significant potential for improving decision-making in health policy, planning and implementation by highlighting unintended consequences.11 It is argued that ‘by replicating the real world in important ways—simplifying where possible while retaining the critical aspects relevant to the problem under study—we can better understand the structural complexity of real-world problems that results from the interaction of specific phenomena and their environments’.12
The introduction of systems science to public health has been met with considerable interest. A plethora of articles now argue for the applicability of systems methodologies and perspectives to a wide range of public health problems, from obesity to tobacco and the social determinants of health.2 ,6 ,13–17 What remains unclear, however, is whether the reality has lived up to this rhetoric. This paper reports on findings from a systematic review designed to investigate the state of systems science research in public health. The objectives were to: (1) explore how systems methodologies are being applied within public health and (2) identify fruitful areas of activity.
Methods
Search strategy
The search strategy and analysis was informed by: the study's aims, previous systematic reviews using qualitative data, and best practice recommendations in the research literature.18 ,19
A systematic review of the literature was conducted by a single research assistant. Searches for peer-review literature were conducted in major databases: ProQuest, Sociological Abstracts, PubMed, Web of Science, Science Citation Index, Social Sciences Citation Index, MEDLINE, Academic Onefile, ScienceDirect, Expanded Academic, EBSCO between 2000 and 2015 (the time in which systems science has gained interest in the field). Search terms included: systems thinking, systems science, complex systems, system modelling, system dynamics combined with public health and key domains of public health activity (ie, obesity, tobacco, alcohol and social determinants of health) (eg, ‘social determinants of health’ AND (‘systems thinking’ OR ‘systems science’ OR ‘complexity science’ OR ‘complex systems’ OR ‘system dynamics’ OR ‘systems modelling’) AND ‘public health’). Articles between 1990 and February 2015 were sourced.
Grey literature searches were conducted using the above terms, with particular attention to major national and international public health conferences including: European Congress on Public Health, International Congress on Public Health, Australian Public Health Association Conference, European Public Health Association Conference, The American Public Health Association Conference. Key exclusion criteria were: articles not published in English, articles which discussed a system (ie, the health system) but were not drawing on ‘systems science’. A list of papers included in the review is provided in online supplementary appendix 1. Owing to limitations associated with using key search terms (within abstracts and titles) to identify literature, we do not suggest that this is an exhaustive list of published work in the field. We do, however, argue that it is a broadly representative sample. As Williams and Hummelbrunner9 note, holism is ‘somewhat of an ideal. In reality, all situations, all inquiries are bounded in some way’.
In total, 341 articles, reports and reviews were identified. A further two were sourced from reference lists. After reviewing abstracts, 124 were excluded on the basis of the aforementioned exclusion criteria and the removal of duplicates. A further 12 were excluded after more in-depth examination (on the basis of the same exclusion criteria). In total, 117 articles were included in the review, with the far majority from peer-reviewed journals (see online supplementary appendix 1 and figure 1).
PRISMA flow diagram.
Articles were categorised by three of the authors (for inter-rater reliability). An inductive qualitative approach to analysis was used. As Dixon-woods et al18 note, systematic reviews can be integrative or interpretive. With integrative reviews, the categories and concepts are set prior to conducting literature searches. With an interpretive approach, concepts and categories arise inductively from the content after the searches are conducted.18 This approach to analysis is otherwise known as an inductive qualitative content analysis.20
Data extraction followed a number of stages. After preliminary categorisation, categories were divided among the authors according to expertise. Each category was reviewed in depth by at least one author and the lead author:
Commentaries, reviews and position pieces—BC, GC;
Analytical framework—BC, GC;
Best practice—AJ, GC;
Modelling—EM, NC, AC, GC.
In reviewing each paper, categorisations were refined. Coauthors provided a detailed summary of each study including its strengths and weaknesses, as well as an overall appraisal of the category.21 ,22 Through this process, the following was systematically extracted from the articles: approach, methodology, transparency, strengths and weaknesses. These were then organised according to theme (ie, commonalities between studies within each category), in order to provide an overview of the state of the field as a whole. The summaries and accompanying analysis forms the basis of the results section of the paper. The assessment of data quality was intrinsic to the goals of the review itself, and therefore, was carried out as part of the analysis.
Results
The literature identified fell into four categories (figure 2): (1) editorials, commentaries and reviews which called for the application of systems science (either broadly, or with regard to specific methodologies or to specific problems) to public health, (2) research which identified systems as a broad analytical lens applied to the research, but which did not draw on systems methodologies or specialist systems insights, (3) research that uses systems methodologies to benchmark best practice in a range of areas and (4) research that claims to have undertaken systems modelling (of different types and differing levels of rigour) (see table 1 for results). Some papers were placed into more than one category; this reflects the fact that there is some ‘blurring’ between the categories identified inductively from the data (ie, the literature sits across a spectrum).
Results of literature searches
Continuum of systems research in public health.
Position pieces
The majority of papers in this category extol the benefits of introducing systems science to public health but contain very little detail about what this would entail. Articles range from broadly advocating systems thinking, to more detailed review articles which tend to use ‘systems science’ to refer to specific aspects of systems or systems methodologies rather than using it as a synonym for ‘multilevel’ or ‘complex’.
The most promising articles in this category offered insights into how systems science might ‘map’ onto existing practices and knowledge in public health, such as prevention science.1 Luke et al,61 for example, argue for systems science in public health identifying specific methodologies in system dynamics, network analysis and agent-based modelling. In doing so, they provide good examples of problems that can benefit from a systems approach, including infectious disease, tobacco and obesity. While these papers hint at the potential of systems science, very few contained details on how this potential could be operationalised.
Analytical lens
Papers in this category claimed to situate their analysis in the field of systems science. For example, using systems concepts to guide the analysis of qualitative data, or incorporating systems concepts into interview schedules in stakeholder analysis. In the latter, this took the form of brainstorming potential systems connections within the research team, and then constructing interview questions to explore these connections.23 As such, this category of papers did not use systems methodologies in the design or collection of data. This meant that systems concepts were generally being used—or superimposed—onto data as a means to recognise the interconnected and interdependent nature of many phenomena. Arguably, many other theories and frameworks that have been well used within public health research already provide this lens (eg, Diffusion of Innovations Theory).79 ,80 The weaker articles in this category claimed to be using a systems lens and/or situate the work within a complexity theory paradigm without any further exposition of systems methodologies, concepts, or how exactly their work is framed, supported or enhanced by a systems approach.25 ,27 ,29
Benchmarking best practice
The papers in this category sought to benchmark or assess the likely efficacy of systems thinking in public health practice, using systems concepts and/or methodologies including soft systems methodology, to underpin intervention and/or evaluation efforts. Here, ‘practice’ is used in the broadest sense—to encapsulate the work of both health promotion and public health practitioners, as well as policymakers. For example, Johnston et al developed a method to benchmark policy recommendations against systems leverage points. In doing so, they highlighted that many recommendations currently made in obesity policy hit ‘weak’ leverage points (where many actions will be required to create substantive change). Others, such as Kwamie et al41 and Prashanth et al44 combined systems concepts with realist approaches to analyse how interventions, such as leadership programmes, had variable outcomes depending on the context into which they were delivered. Research on the implementation of systems-type interventions has shown that without training and clear guidelines on systems-based implementation, then practitioners resort to their previous experience in delivering multicomponent reductionist-style interventions.45 Arguably, this is the systemic issue that is slowing, or muddling, public health's attempts to engage with systems-based approaches.
A number of papers outlined how systems concepts could be used to enhance and refine policy and practice. Bar-Yam used a systems science perspective to contend that there needs to be two different organisational forms in health services.33 First, to develop organisational systems and structures for tasks that are highly routinised such as immunisation and other forms of preventive health, and to develop organisational systems for tasks that are highly complex and unique. The paper contrasted the way the system is funded and managed in large-scale processes (funding flows between employers, insurers and providers) compared with the highly complex individual decisions required of individual patient care. It was commented that efforts to routinise and regulate care will always diminish effectiveness, given that optimal outcomes are based on highly individualised approaches. Johnston and Finegood39 used their systems leverage points framework, titled ‘intervention-level framework’, to discuss the potential benefits and limitations of public–private partnerships in addressing obesity and non-communicable diseases. MacLean et al42 provided a conceptual framework for vertical and horizontal connections to address childhood obesity. Using a Canadian case example, they were able to demonstrate the connections between different layers of government (vertical integration), and between sectors such as education and health (horizontal integration). It was commented that the model elucidated key partnerships across system elements, but a weakness in the model is that it does not account for dynamic system properties that could account for how and where change occurs in the system.
Only two papers described and commented on initiatives that were specifically designed to be systems-based practice. BeLue et al34 described how a community-based participatory research approach was enhanced by applying the soft systems method causal loop diagramming to understand factors influencing underage drinking in their area. A more critical study by Sautkina et al45 using key informant interviews, discovered that despite an explicit effort to introduce systems thinking in particular settings, there was little evidence of an understanding of specific systems science approaches and very little evidence of its application among practitioners and policymakers. Rather, a number of risk factors were identified, and multiple interventions implemented to address these risk factors. It was concluded that the policy narrative was not very clear about what a systems-based approach meant, and thus, there was no clear direction in how this could be implemented. In the absence of clear guidance, local teams reverted to past experience. This failure was also noted as a symptom of the literature, where there is considerable focus on developing systems-thinking concepts, but very little attention on the attributes of a system-level intervention, and how it could be delivered and evaluated. BeLue et al34 also commented that further training and support in systems-thinking techniques would be required to enhance community capacity.
Interestingly, when assessing the quality of practice in a range of settings, success was generally defined as the presence of systems-thinking practice, irrespective of the quality of this practice (although it was noted that efforts at increasing systems-thinking capacity were required to improve practice).34 While the critique of how systems concepts are used in evaluation practice is more developed than critiquing the use of systems concepts in implementation practice, reviews of evaluation practice also focus more on understanding when and in what way systems-based concepts are being used rather than any analysis of whether these concepts are improving evaluation practice.32 ,47 Thus, while some interesting methodologies are beginning to develop in this area, at this stage it appears that researchers are primarily assessing when and where systems practice and evaluation are occurring rather than appraising the utility of systems practice and evaluation itself. These researchers conclude that more practical guidelines, training and partnership with experienced systems modellers are required if we are to see systems concepts applied more broadly in public health practice and evaluation.
System modelling
Papers categorised as ‘systems modelling’ used a range of qualitative and quantitative methods to model various systems or subsystems relevant to public health problems. When assessed as a whole, many studies in this category exhibited serious weaknesses. Notably, public health research that uses systems-modelling techniques often fails to assess whether the models constructed therein are any good. In the most advanced articles, such as the well-known Forsight maps of obesity,5 the methods used to build the model were explained in rigorous enough detail that an assessment of process could be made. That is, whether the process used to build the model seemed sensible, likely to produce useful results, and is consistent with accepted methodologies within the system dynamics discipline. However, any actual validation of the model thus constructed was rare. Often, this was due to a paucity of data, or the difficulty in generalising across social/geopolitical boundaries.
A number of papers stood out as examples of best practice in system dynamics research for public health36 ,54 ,58 ,59 (as opposed to best practice within the field of systems science more broadly, which is outside the scope of this review). From these examples of best practice, criteria can begin to be constructed for reporting the results from system dynamics modelling in the health domain. For example,
When developed through third-party software, the model should be made available for download and linked in the paper or an online supplementary appendix (as done by Fallah-Fini et al54).
Implementation of the model should ideally be done via open source or widely used software which has thorough documentation (this would enable other researchers to assess the quality of the differential equation solvers used). In some instances, even where software was named, it was inaccessible and quality could not be checked.50
Preferably, the model equations and variable relationships should be spelled out explicitly in the paper (as seen again in ref. 54). This is especially the case where the model itself cannot be made available for whatever reason.
There should always be some attempt made to validate results except possibly in exceptional circumstances.
In the remaining weaker papers, there was often not enough detail presented to understand how the model was constructed.5 ,14 ,35 ,36 ,69 Often, the mere existence of the model itself was presented as sufficient justification of the work that had gone into the process.5 ,69 It was often asserted that policy recommendations should be built on this model not because the model itself was useful or accurate, but because the process that produced it was alleged to be of greater validity than traditional approaches simply by having been built up within a systems-based framework.16 ,70 Yet, very few of the weaker articles sufficiently elaborated on the model-building process.
Discussion
Lich et al1 contend that despite a growing appreciation for ‘multiple levels’ and systems of influence, public health is yet to take full advantage of the analytical approaches—or toolbox—provided by systems science. Our systematic review of existing public health research that utilises systems approaches supports this claim. In particular, the fact that close to half the papers identified by the review process are commentaries or calls for the application of systems methodologies, but do not use systems methodologies. On the basis of this finding, we focus our discussion on areas that can be progressed. The review found two areas of public health systems activity worth reflecting on more deeply: systems modelling and the use of systems science to benchmark best practice.
Systems modelling is often treated as the area of systems work that holds the most promise for solving public health problems, and particularly, for policy.9 ,12 ,13 ,33 ,77 Our review of the existing modelling in the field, to date, highlights cases of best practice, though it also suggests that a number of methodological weaknesses need to be addressed.
First, in terms of weakness, the review raised questions regarding accountability. The nature of public health means that social dynamic models will always be fluid, subjective and non-holonomic (ie, the model will depend on the process as much as the fundamental social conditions/data). However, this does not preclude accountability. If the systems paradigm for policy creation is to be taken seriously, a minimum standard of accountability and repeatability needs to be adhered to by researchers. In other words, there needs to be adequate space in published articles given over to the statistical and heuristic methods used to build up the model. As noted previously, at times, the mere existence of the model itself was presented as sufficient justification of the work that had gone into the process, and it was asserted that policy recommendations should be built on this model not because the model itself was useful or accurate, but because the process that produced it (a process that was often obscured or glossed over) was thought to be of greater validity than traditional approaches, simply by virtue of having been developed from a dynamic systems perspective.16 ,37 ,70
We advise a higher expectation regarding validation and verification practices for the dynamic systems models published in the public health-related discourse. This would allow informed readers to replicate and adapt, or improve on the modelling process. This should occur prepublication, but if sufficient data is not available then authors should, at a minimum, present a roadmap for how non-verified models might be empirically validated. This would enable future researchers to assess the efficacy and adequacy of the models, and suggest improvements or refinements. Further, public health practitioners would do well to link with the established field of system dynamics to find methods that have been developed within the discipline over the past 60 years—where academic debate and peer review have strengthened the approach.50 ,81–90
The second area to emerge from the review which is worth deeper discussion is the use of systems science to benchmark best practice (whether in policy or health promotion). This has produced some innovative methods and approaches.39 ,45 For this area to be effective, research and implementation must go beyond assessing whether systems concepts are simply in use, and instead, consider the quality and effectiveness of systems-based practices and approaches. This requires greater research attention on the attributes of a system-level intervention and how it could be delivered and evaluated, in addition to the current focus on development of systems-thinking concepts.45 Within the ‘softer’ systems methodologies, such as soft systems methodology, and the use of systems-based methods for benchmarking good practice, the existence of a plurality of methods poses a challenge for the field, which strike a balance between allowing for inclusive and innovative uses of systems-based approaches that are ‘fit for purpose’, while simultaneously maintaining scientific and methodological rigour.
Overall, our review emphasises that systems-based approaches should not be thought of as the ‘new way forward’; they should be thought of as an additional set of methods to organise and analyse information about complex and dynamic public health phenomena. Systems-based approaches can sit alongside other public health approaches and bring a unique contribution to the field of research and practice, as highlighted in this review. Further, it is not only the methods of systems science that may be useful, but also their methodological positioning.
We use, for example, the methodological positioning of soft system methodology. Soft system methodologies have long been underpinned by the notion of a ‘system as a metaphor’ for understanding and communicating about complex phenomena.83 ,91 Conceptualising systems in this way allows the researcher to use notions in systems thinking, such as accumulation, feedback and endogenous behaviour as metaphors for facilitating knowledge transfer, thereby creating and/or revealing integration and comparison between the worldviews of system actors. Dyball and Newell83 call such metaphors ‘powerful ideas’, as they provide a way to build a shared understanding between people of different disciples and of repeated dynamics between contexts. The metaphors within systems science are simultaneously generic enough to maintain their applicability in a number of different contexts such as climate change science, urban planning, or the study of health inequalities, but accurate enough to elucidate the structure of complex problems.
These methodological positions align well with the goals of research in our ‘benchmarking best practice’ category. This suggests that systems concepts can act as useful metaphors for enhancing our integrative practice, whether as health promotion practitioners or policymakers. Using these metaphoric concepts as a basis for conceptualising complex health phenomena also negates the need to overcome the extensive and expensive technical hurdles of undertaking dynamic systems modelling. This is particularly relevant to highly complex and politically sensitive areas of public health research such as the social determinants of health.
It is also worth noting that the evidence produced by different systems science methodologies and methods hold different value to policymakers (or researchers), depending on the epistemological learning. Within the systems-based field, a distinction is broadly made regarding ‘hard’ versus ‘soft’ systems methodological approaches. While this distinction is not absolute, in public health dynamic systems modelling may be more appealing to those who hold positivist epistemological positions (ie, where data-driven models provide decision makers with concrete evidence of where and how to act). However, policy scholars increasingly see policy as emerging from policy networks. This body of work perceives policy as being shaped by ‘networks that are fluid, constantly changing structures which can be shaped by the agency of actors within them as well as by external and temporal constraints’.94 This does not mean that research evidence (in modelling form or other) does not inform policy actors, but that it is not an instrumentalist/transactional arrangement, and that the shifts it creates will not be transformative or radical.92–94 Increasingly, public health is moving away from instrumentalist views of policy—with a growing backlash against the evidence-based policy paradigm.95–97 Hence, if one of the promises of systems methodologies for public health is their utility for decision makers and policy actors, a closer analysis is needed of how public health conceptualises policy change, and the ways in which systems science can and cannot feed into this. For example, the models produced are not necessarily widely applicable across diverse contexts with different cultural dynamics.98
Finally, it is worth reflecting on the need to ask the right questions of systems science. This review has highlighted some of the rhetoric around systems-based approaches to public health. Systems-based approaches and complex systems science should not be framed as an unsung solution to all the major challenges in public health. Indeed, making a system-dynamics model does not give policymakers agency in spaces where they currently have none. Public health problems are already deemed complex, and systems-based approaches can contribute to changing the language, methods and methodologies for conceptualising and acting within this complexity. In order for systems-based approaches to live up to their ‘rhetoric’, the public health and prevention field must ask the right questions of the discipline, and not expect systems-based methodologies to provide the ‘silver bullet’ answers to some of our biggest challenges, such as preventative action on the social determinants of health. Increased literacy as to the forms of evidence that different systems-based methodologies and methods can produce will aid the public health and prevention field to ask the right questions of systems science.
Conclusions
Overall, our systematic review of the systems science literature in public health revealed that there is a great deal of interest in how systems concepts and approach can aid public health. Our analysis suggests that soft systems modelling techniques are likely to be the most useful addition to public health, and align well with current debate around knowledge transfer and policy. It is also important to note, however, that the full range of systems methodologies is yet to be engaged with by public health researchers and practitioners.
Limitations
It is possible that there are studies that were not turned up in the review, particularly if reported in the grey literature. However, we believe the sample derived from our review process represents the field in its current state.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors GC conceived of the study. GC, BC, NC and AJ created the search protocol. All authors contributed to analysis, drafting the paper and the conclusions.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.
↵i This paper does not provide an overview of systems theories or methods. These are provided in many of the papers reviewed, and comprehensive introductions can be found elsewhere.9 ,10