Skip to main content

Lessons for introducing stakeholders to environmental evidence synthesis

Abstract

Involving stakeholders in systematic reviews is common practice and is advised in the Collaboration for Environmental Evidence (CEE) Guidelines (v.4.2). Frameworks for engaging stakeholders exist and should be used; however, there are additional lessons to be learned in a country, or region where evidence-based environmental management is an emerging paradigm. Based on our experience working with Canadian governmental institutions, we provide five lessons that we have learned while introducing stakeholders to the CEE systematic review (hereafter SR) process. These lessons are: (1) Advocate for a systematic review with broad geographical scope and target audience; (2) Control stakeholder mission-creep; (3) Establish a mutually beneficial timeline; (4) Reduce the potential of biased targeted searches; and (5) Manage stakeholder expectations. By incorporating these lessons into existing frameworks, we hope to make the introduction of SRs to stakeholders more efficient to conserve resources and maintain long-lasting, productive relationships between the review team and stakeholders.

Background

An important component of conducting systematic reviews (SRs) is the engagement of stakeholders, which can include subject matter experts in academia, non-government organizations, and government, or anyone with a stake in the findings of the review [1, 2]. Stakeholder involvement in environmental management has been reviewed [3] and frameworks for involving stakeholders in SRs have been developed [4, 5], most recently in environmental management [6]. These frameworks should be the basis for stakeholder involvement; however, certain challenges can still arise when stakeholders are participating in the process for the first time. Typically, we adopt a common definition of stakeholder as defined in Haddaway et al. [6] in that it includes “all parties that may affect or be affected by a review”; however, for the purpose of this commentary, our advice applies mainly to those involved in the review process from question definition to review synthesis (i.e., commissioners, funders, advisors).

As is the case in many countries, in Canada the concept of formal evidence synthesis in environmental management is an emerging one [7]. Canadian government institutions, such as Parks Canada and Fisheries and Oceans Canada, have a history of using evidence-based advice, although they have only recently begun commissioning formal Collaboration for Environmental Evidence (CEE) SRs to integrate into their decision-making processes. This new relationship does not come without its challenges and provides an opportunity to develop a process that benefits both the stakeholders and the review team. Our authorship team has experience conducting five environmental SRs and writes this from the Canadian context where currently, most SRs are commissioned by government institutions seeking to address predefined management topic(s). With this in mind, we provide five lessons to consider when undertaking a SR with stakeholders that are new to the process to help ensure a successful relationship between the review team and the stakeholders involved.

Advocate for a systematic review with relevant geographical scope and target audience

Those that commission (and fund) systematic reviews in the environmental sector often are interested in questions that have an inherent regional or national focus. However, the literature relevant to a given SR may be much broader such that it is worthwhile and indeed necessary to include data from other jurisdictions. It is important to engage with relevant stakeholders when developing/refining the “question(s)” and in determining trade-offs with a particular scale—local specificity or broader applicability. When a SR is done without a particular jurisdiction as the focal area, the SR inherently has broader value and relevance to the international community. For example, it would make little sense to conduct a SR on the effectiveness of habitat restoration activities at the level of the province or state in North America. Instead, it may make sense to approach things on an ecoregional scale (traversing multiple jurisdictions or even countries) or even a taxonomic perspective (e.g., salmonids). Details regarding the scope of the SR and associated search should be discussed during the commissioning phase but needs to be decided such that the funder feels the work will be relevant to their needs. It is important to be sensitive to the fact that a funder (e.g., a government agency) may not want to “spend money” on examining literature from the other side of the world. Yet, some species and ecosystem types (or ones quite similar—e.g., the congeneric yellow perch in North America and European perch in Europe) occur in other jurisdictions such that it is sensible to consider diverse literature. In some cases, even if the species or ecosystems are quite different, the issue/topic is germane (e.g., fish removal from lakes), which again points for the need to not impose jurisdictional limitations on SRs unless there is good reason for doing so. This also applies when the question represents a local knowledge gap and necessitates a broader geographical scope to uncover relevant evidence. Issues regarding the external validity of studies tends to increase concomitantly with the breadth of literature considered. Through discussions and use of examples it is important to consider the potential value of including the relevant global literature even when engaging in an SR that was triggered by a local issue. By doing so the proponent will benefit but so will the broader scientific and management community as the global relevance (or at least beyond a single jurisdiction) of the work is realized. However, the ultimate decision on scale should be up to the commissioner of a SR.

Control stakeholder mission-creep

Constructing a clear, carefully articulated question for a SR is a crucial step, and often necessitates a compromise between comprehensiveness and detail [1]. This is especially true in environmental science, where interacting processes can quickly make a problem very complex. For example, a systematic review that examines the impact of an environmental stressor may need to consider interactions with other stressors (which frequently co-occur), as well as the mitigating influence of varying baseline conditions. In addition, given the ever-growing body of literature, there is a clear risk that an overly comprehensive question can make a SR intractable, especially in the timeframes often required by government agencies (e.g., a hard deadline at the end of fiscal year; see lesson 3 below).

In contrast with a tendency for some individual stakeholders to adhere to a strong local focus (see lesson 1 above), there is a tendency among broader stakeholder groups toward ‘mission creep’. In our experience, government agencies tend to be more hierarchical than research institutions that may be actually conducting the SR. Involving several stakeholders in question design, means involving their managers, and their managers’ managers as well. Governments are also subject to (sometimes rapidly) changing priorities. These issues can lead to a proliferation of opinions, and an understandable tendency toward including as many requests as possible into the question framework. If all stakeholders up the chain of command do not have a basic understanding of the strict protocols and rigour of SRs, there can be pressure to design a review with an overly-broad or vague question that attempts to string together many disparate elements. Open lines of communication can help to alleviate this problem, and balance the needs of stakeholders with the requirements for conducting a rigorous SR within the allotted timeframe. Involving both coordinators and implementers during the development of the review protocol can be a strength of government-supported SR, and it is always appreciated when busy managers and practitioners take an interest in the review process. This is worth the hazard of mission-creep, but makes it crucial to provide an initial, clear description of the SR process, and what distinguishes it from traditional literature reviews.

Establish a mutually beneficial timeline

During the early stages of SR planning, it is important to discuss and agree on a realistic timeline with those commissioning the review. Depending on the experience of the review funders, it may be beneficial to provide some form of training or at the very least discuss the process thoroughly to communicate a reasonable timeline for the review. Stakeholders that have previously commissioned a SR may have a better appreciation for the time required to ensure a comprehensive review. In certain cases, some flexibility in the process may be necessary. This is particularly the case with government funders that have fiscal or other internal deadlines as the length of time SRs take will often cross one or more fiscal years. In this case, it may be useful for the funders to introduce deliverables that are independent of the CEE process and satisfy their accounting needs. An example of this is the need to provide a narrative report of the literature base in the form of review descriptive statistics once screening is complete to report on the size of the review, or alternatively an interim systematic map report. Deliverables may also include giving presentations, webinars, or holding update meetings. Deadlines and deliverables like this are best discussed and agreed upon early in the process so those leading the SR can adjust timelines accordingly. It may also be helpful to re-confirm these deliverables throughout out the process whether at the beginning of meetings or through a more formal project update, to ensure everyone involved has clear expectations.

Reduce the potential of biased targeted searches

Often the search strategy of a SR includes conducting targeted searches for the more difficult to find, grey literature. These sources of information can be identified through consultation with: (1) stakeholders, (2) the network of experts leading the SR, and (3) the broader community via social media and email solicitations. If not executed well, individually-driven searches can lead to bias in the derived literature set, if for example, researchers leading the review put more effort in targeting particular geographical regions (e.g., jurisdictions or even countries) or researchers. From our experience, ways to reduce the potential of biased targeted searches can occur at different stages of the SR process. For example, at the early stages of the SR, to reduce the potential for biased advice on where to uncover new sources of information, we recommend forming an advisory team made up of stakeholders and experts that is both geographically balanced to the extent possible, and diverse in regards to their backgrounds, views, and skill sets (e.g., academics, practitioners, topic experts, SR experts, etc.). Additionally, at the stage when these targeted searches are being undertaken, it has been our experience, that stakeholders and/or the broader community can apply pressure or encouragement to pursue information in one or a few narrowly focused direction(s)—especially if the advisory team is geographically unbalanced and/or with similar backgrounds etc. as per our previous comment. Furthermore, this potential issue can be magnified if these narrowly focused directions are suggested late in the review process, not leaving sufficient time to adequately follow up on all potential sources of information. These situations require a discussion to ensure that the suggested sources of information will be considered, but only if effort is equally weighted across all identified sources of information.

Manage stakeholder expectations

Two common expectations we have encountered when introducing stakeholders to evidence synthesis are that: (1) all research retrieved will be included in all stages of the review, including narrative and/or quantitative synthesis, and (2) a quantitative synthesis will be conducted. We have learned that frequent communication between the project leaders and those commissioning the SR is essential at all stages of the review process to set and manage expectations for the final review product. Of particular importance for those leading the review, is the need to explain to the SR commissioners that even if it is determined that there is sufficient research on the given topic, this does not mean all of the existing relevant research is of the same quality or in fact “usable”. For example, studies may contain particular deficiencies in the design, conduct, or analysis, such that the study has high susceptibility to bias, and therefore may need to be excluded from narrative/quantitative synthesis. Also, how the data are reported can limit quantitative analysis if insufficient methodological details were provided (e.g., means, variability, sample size) [8], or if the study had insufficient replication to allow for effect size calculations [9]. In reality, many of the identified sources of information are excluded at the later stages of a review and this is not always evident from the outset. Therefore, those leading the review may need to explain from the beginning, and continually remind stakeholders, as to the objectives and benefits of SRs over other forms of literature reviews. Indeed, it is the quantity and quality of the existing literature that dictates the final review product, and as such, expectations need to be managed early on, and throughout the review process to ensure the proponent is satisfied with the final product. One further approach we are using to help set and manage expectations is to provide training in the form of workshops for the commissioners to introduce the role of SRs in evidence-based management and to familiarize them with the steps in the SR process by explaining what each step involves and why it is important. Ideally this would be provided prior to the start of any commissioned SR.

Conclusion

Being engaged in environmental evidence synthesis in Canada where this activity is relatively new has taught us valuable lessons through engaging stakeholders that are unfamiliar with the CEE process. We share these lessons to provide others who are planning on conducting CEE SRs in a similar situation with advice on overcoming some of the inherent challenges. Ultimately, a common thread among these lessons is the importance of clear bi-directional communication between the review leaders and the stakeholders throughout the SR process. We have found that beginning each stakeholder meeting with a refresher on the process, as well as updates (including volumes of literature encountered, and the time taken at each step), have helped government agencies appreciate the value of SRs and what distinguishes them from a traditional literature review.

By incorporating these lessons into existing frameworks, we hope to make the introduction of SRs to stakeholders more efficient in order to conserve resources (i.e., time, money), manage expectations, and maintain long-lasting, productive relationships.

Abbreviations

CEE:

Collaboration for Environmental Evidence

SR:

systematic review

References

  1. CEE. Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. The Collaboration for Environmental Evidence, 4.2. 2013.

  2. Freeman R. Stakeholder management: a strategic approach. New York: Pitman; 1984.

  3. Reed MS. Stakeholder participation for environmental management: a literature review. Biol Conserv. 2008;141:2417–31.

    Article  Google Scholar 

  4. Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, Tunis SR. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. 2012;1:181–94.

    Article  Google Scholar 

  5. Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in systematic reviews: knowledge transfer for policy and practice. J Contin Educ Health Prof. 2008;2:67–72.

    Article  Google Scholar 

  6. Haddaway NR, Kohl C, da Silva NR, Schiemann J, Spök A, Stewart R, Sweet JB, Wilhelm R. A framework for stakeholder engagement during systematic reviews and maps in environmental management. Environ Evid. 2017;6:11.

    Article  Google Scholar 

  7. Cooke SJ, Rice JC, Prior KA, Bloom R, Jensen O, Browne DR, Donaldson LA, Bennett JR, Vermaire JC, Auld G. The Canadian context for evidence-based conservation and environmental management. Environ Evid. 2016;5:14.

    Article  Google Scholar 

  8. Haddaway R. A call for better reporting of conservation research data for use in meta-analyses. Conserv Biol. 2015;29:1242–5.

    Article  Google Scholar 

  9. Haddaway NR, Verhoeven JTA. Poor methodological detail precludes experimental repeatability and hampers synthesis in ecology. Ecol Evol. 2015;5:4451–4.

    Article  Google Scholar 

Download references

Authors’ contributions

The manuscript was drafted by JJT, TR, JRB, and SJC. All authors read and approved the final manuscript.

Acknowledgements

Authors would like to thank Neal Haddaway for the invitation to contribute to the special issue, and for providing helpful feedback on this manuscript. We would also like to thank all stakeholders that have contributed to our ongoing SRs for being a part of this learning process as we build capacity at the Canadian Centre for Evidence-Based Conservation and Environmental Management. We also thank the reviewers for their constructive comments.

Competing interests

The authors declare that they have no competing interests.

Availability of data material

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Funding

Natural Sciences and Engineering Research Council of Canada Discovery Grants to Cooke and Bennett. Canada Research Chair in Fish Ecology and Conservation Physiology to Cooke. Carleton University Research Excellence Fund to Cooke.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jessica J. Taylor.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Taylor, J.J., Rytwinski, T., Bennett, J.R. et al. Lessons for introducing stakeholders to environmental evidence synthesis. Environ Evid 6, 26 (2017). https://doi.org/10.1186/s13750-017-0105-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13750-017-0105-z

Keywords