A five-step approach for stakeholder engagement in prioritisation and planning of environmental evidence syntheses

Abstract

Systematic reviews and systematic maps, regarded as a gold standard for syntheses of documented research evidence, are increasingly used to inform decisions in environmental management. To increase their relevance and uptake, systematic reviews and maps can be planned with the help and engagement of stakeholders, i.e. organisations and individuals involved in and affected by environmental policy-making and practice. We report on the emprically tested five-step approach that the Mistra Council for Evidence-based Environmental Management (EviEM) is using to engage stakeholders and incorporate their views and opinions in the prioritisation and planning of reviews, including (1) stakeholder identification; (2) identification of policy- and practice-relevant topics; (3) framing and prioritisation of review questions; (4) establishment of the specific scope of a review; and (5) a public review of a draft review protocol. We provide examples from EviEM’s reviews and describe various challenges and valuable lessons learnt from the engagement process, hoping that this will be useful reading not only for reviewers, but also for stakeholders who plan to participate in the engagement process.

Background

Systematic reviews and systematic maps (here also referred to as ‘reviews’) are regarded as a gold standard for syntheses of documented research evidence, and they are increasingly used to inform decisions in environmental management [1, 2]. Following core principles of transparency, objectivity and repeatability, they aim to identify, collect and synthesise available evidence, attempting to minimise subjectivity and bias at each stage of the review [3]. To be valuable for environmental policy and practice, evidence syntheses must address relevant questions, and their findings need to be recognised as legitimate evidence. One way to increase their value is to engage with organisations and individuals involved in and affected by environmental policy-making and practice [4,5,6]. Whilst there is growing evidence that stakeholder engagement can help reviews become clear, relevant, broadly communicated, and used in policy and practice [5, 7], the roles of stakeholders in the engagement process have varied across review teams and topics [8]. We define stakeholders as those who use or may be affected by review findings [9], including researchers and subject experts, practitioners, commissioners, and representatives of governmental and non-governmental organisations [10]. Stakeholder engagement has been defined as a ‘bi-directional relationship between the stakeholder and the researcher that results in informed decision-making about the selection, conduct, and use of research (findings)’ [11], p. 986.

Systematic reviews and maps are conducted through a step-wise process including: (1) establishing a review team; (2) formulation of a question and scope, involving a scoping exercise; (3) peer review and publication of a review protocol; (4) searching for evidence; (5) screening of evidence for eligibility; (6) critical appraisal of study validity (not obligatory for maps); (7) meta-data extraction; (8) data extraction (in systematic reviews only); (9) synthesis (in systematic reviews only); (10) reporting and communication of the review findings [3, 12, 13]. Stakeholders can be engaged throughout the whole review process [14], but here we describe engagement with stakeholders in prioritisation of review topics and in the review planning stage, and as an example we use the Mistra Council for Evidence-based Environmental Management (EviEM) approach. EviEM is the Swedish centre in the network of the Collaboration for Environmental Evidence (CEE), a coordinating body for promotion, conduct and registration of environmental systematic evidence syntheses. CEE hosts a library of reviews and protocols, produces review guidelines and ensures that registered reviews comply with the rigorous review standards [3, 15]. Since 2012, EviEM has conducted systematic reviews and maps relevant (but not restricted) to Swedish environmental policy and management [16]. During 2012–2017 EviEM has been funded by the Swedish Foundation for Strategic Environmental Research [17] and governed by an independent Executive Committee comprised of scientists, evidence synthesis experts, and stakeholder representatives. EviEM is financially and politically independent. It has a secretariat with methodology experts (project managers) who conduct systematic evidence syntheses with the help of international scientific experts. Unlike reviews entirely driven by commissioners and their particular interest in a certain topic, the findings of which may have limited generalisability [9, 18], EviEM reviews are intended to be ‘public goods’ [19]. They are placed in an open domain, available to a global audience and have relevance for a broader range of stakeholders.

EviEM uses a relatively formal but flexible approach to stakeholder engagement in the review prioritisation and planning stages. To incorporate stakeholder views and opinions, EviEM initiates a five-stage process: (1) identification of stakeholders; (2) identification of policy- and practice-relevant topics; (3) framing and prioritisation of review questions; (4) establishment of the specific scope of a review; (5) a public review of a draft review protocol. We describe details of these five stages in the following sections (see also Fig. 1).

Fig. 1

A five-step approach for stakeholder engagement in prioritisation and planning of evidence syntheses

Stakeholder identification

Stakeholder identification is critical to the entire stakeholder engagement process and review conduct [10]. However, it may be difficult to know who the stakeholders are and to identify a representative stakeholder group [20]. EviEM identifies stakeholders at two different levels and for two different purposes.

First, to understand knowledge needs (Fig. 1, stage 1), EviEM identifies a broad range of stakeholders across the whole environmental sector in Sweden. This is done through a detailed stakeholder analysis, partly based on methods by Schmeer et al. [21]. In short, EviEM conducts a search for stakeholders among representatives of the Swedish Parliament, various Ministries, county and municipality administrations, governmental agencies, research funding councils, Swedish and international non-governmental organisations, industry representatives, and relevant European policy makers. The next step is to get to know stakeholders’ characteristics, their main interests and roles, level of influence, fears and expectations, and possible links to EviEM. Finally, based on all collected information, stakeholders are classified according to their priority (low, medium, high, or critical), as seen from EviEM’s perspective.

Second, to refine review questions, a specific group of stakeholders is identified for each review, and this process is described in “Framing and prioritisation of review questions” section.

Identification of policy- and practice-relevant topics

To identify policy- and practice-relevant topics, EviEM arranges meetings with stakeholders across the entire environmental sector, inviting them to state their needs for knowledge. Meetings typically start with a short introduction to systematic evidence synthesis methodology, after which participants discuss potential review topics. The main outcome of these ‘general’ stakeholder meetings is a list of topics and questions that usually includes broad global, national or regional environmental issues, perceived gaps in the evidence-base, and controversial environmental questions recently discussed in public debate (Fig. 1, stage 2). Two examples of stakeholder-generated topics suggested to EviEM are “How is biodiversity affected by anthropogenic interventions in shallow bays (such as removal of algae, dredging and embankments)?” (later reformulated into a more focused question that is now being reviewed [22]) and “What are the reasons for the decline of sea birds in the Baltic Sea region?”. Questions proposed by stakeholders are often more suitable for systematic mapping than for systematic reviewing. For example, they are often open-framed, i.e. not specific enough to be answerable in a single study and therefore not possible to answer in a synthesis of similar studies [3]. It could also be that a synthesis of the evidence is not needed because the question has already been addressed by recent reviews, or that it is not feasible, e.g. because there is a lack of primary research on the topic.

Framing and prioritisation of review questions

The next step is to rephrase the questions if needed and prioritise them in collaboration with stakeholders. Several priority setting initiatives in other research areas, especially medicine, have developed a number of approaches to accomplish this critical work (see e.g. [23, 24]). At this stage (Fig. 1, stage 3), EviEM review experts screen proposed topics to determine whether they are reviewable as such, or whether they should be split up or narrowed down into one or more specific questions. As part of this question-framing process, EviEM undertakes scoping studies of proposed review topics (e.g. [25,26,27,28,29,30]). Scoping studies are summaries of the volume of existing evidence on a specific topic. They introduce the review topic, investigate if any other (systematic or traditional) reviews on the same topic already exist, seek to clarify whether there is sufficient scientific literature and need for a systematic review or map on the topic, and identify review-specific stakeholders [31]. Review-specific stakeholders are mainly identified through ‘snowball sampling’ [32, 33], usually starting with relevant stakeholders identified in a broader context (see “Stakeholder identification”) who may be able to refer to other, less visible stakeholders. Snowball sampling may entail community bias through overrepresentation of certain stakeholders and their interests [32]. However, EviEM strives to minimise that risk by active searches for stakeholders with different or opposing interests.

When conducting scoping studies, EviEM may also engage scientists working with the topic to ensure that the review question is scientifically meaningful, i.e. answerable, conceptually clear, and methodologically feasible. One or more of these scientists may later be recruited to the team that will conduct the review. Even where a stakeholder-generated question seems to be specific enough for systematic reviewing or mapping and there is no need for additional question-framing, it may still be difficult to conduct a review due to methodological constraints. For example, in 2012 EviEM received a suggestion to review adverse effects of perfluorinated alkylated substances (PFASs) on marine biota. At that time, systematic review methodology in toxicology and chemical risk assessment was less developed than it has become more recently (see e.g. [34,35,36]), and the EviEM Executive Committee concluded that validation of the review methods would require significant efforts. One of the difficulties with the suggested review question was to define the outcome. However, questions related to PFASs were considered important to address, and EviEM therefore invited both stakeholders and scientists (topic experts) to a focus group to identify a reviewable and scientifically meaningful question that could also have relevance for the stakeholders. The question finally selected was how PFAS phase-outs have affected PFAS concentrations in the environment. In this case, the outcome was fairly easy to define, and the question is currently being reviewed [37].

If multiple reviewable questions are identified during the scoping process, key stakeholders (mainly those who originally suggested the topic) may be asked to prioritise the most relevant ones. Based on the conclusions of scoping studies and stakeholder prioritisation, the review experts propose specific questions for systematic reviews or maps to the EviEM Executive Committee, which takes the final decision regarding the selection of questions to be reviewed. To guide the Committee in their decision, mandatory and optional criteria for EviEM review questions have been established. The mandatory criteria stipulate that a question should (1) deal with conditions in the natural environment, (2) be relevant to the situation in Sweden, (3) be well-defined, conceptually clear and reasonably limited in scope, (4) deal with problem descriptions or countermeasures whose scientific support is insufficient, disputed or incompletely known, and (5) be covered in the scientific literature (or by other investigations) to such an extent that a systematic review or map could be undertaken [31].

Establishing the specific scope of a review

Once a review question has been approved by the EviEM Executive Committee, stakeholder engagement resumes through a ‘review-specific meeting’ (Fig. 1, stage 4) led by one of EviEM’s review experts and a scientist (topic expert) appointed to chair the review team. At this meeting, representatives of all stakeholders identified by the scoping study are invited to discuss the review question in detail. Through an open dialogue, the participants are encouraged to share their views and help refining the scope and focus of the review by specifying preferred PICO/PECO elements of the review question (i.e. population, intervention or exposure, comparator, and outcome) [3, 38] and criteria for inclusion of studies. The stakeholders are also involved in the development of a search strategy by suggesting search terms and sources of relevant literature (the latter is especially important for locating grey literature). Stakeholders not able to attend the meeting are invited to send their comments by email. Useful stakeholder suggestions are incorporated in a draft review protocol subsequently written by the review team.

In this process of knowledge exchange the stakeholders can improve the relevance of EviEM’s reviews as they fine-tune the review scope according to their priorities. The general topic of a review cannot be changed at this stage, but discussions during stakeholder meetings often lead to either a broadening or a narrowing of the scope of the review. For example, during preparation of the protocol for a systematic review on the ability of wetlands to remove nutrients from water [39], at least two important study inclusion criteria were changed following advice from the stakeholders. The scope of the review was extended to cover removal of phosphorus as well as nitrogen, whereas it was narrowed down to cover created and restored wetlands only (excluding natural wetlands).

Different stakeholders may have different priorities, and all their suggestions may not be equally feasible or scientifically sound. The final decision regarding the scope of a review is always made by the review team when writing the protocol. However, to avoid situations where some stakeholders may feel that their suggestions were not considered, justifications for the final decisions are always communicated back to the stakeholders or provided in the review protocol (see “Public review of a draft review protocol”).

Public review of a draft review protocol

When a review protocol has been drafted by the review team, it is published on the EviEM website and opened for a public review (Fig. 1, stage 5). A review protocol is a detailed methodological plan for the conduct of a review, explaining rationale, review question and methods for all stages of the review process [3]. Anyone interested is welcome to comment on the draft, but all previously identified stakeholders receive a special invitation to continue their engagement. After a period of 2 to 3 weeks, the public review is closed, and the protocol is revised based on comments received.

Stakeholder comments received during the public review process can modify the scope of a review as well its applicability and relevance. For example, EviEM’s ongoing review of roadside management [40] was initially intended to cover management effects on vascular plants and all kinds of animals, but when the protocol was open for public review, one stakeholder pointed out that studies of mammals and birds along roadsides may often be difficult to use as evidence of management effects. After careful consideration, the review team decided not to include effects on vertebrates in the review, but to focus on vascular plants and invertebrates instead.

The development of the protocol, including the engagement of stakeholders, is reviewed by the EviEM Executive Committee. After their approval of the process, the protocol is submitted for peer review, and this marks the closing of the early stakeholder engagement in prioritisation and planning of evidence synthesis.

The Knowledge project

Although stakeholder engagement is depicted in Fig. 1 as a step-wise process, working with a range of stakeholders on developing the scope or focus of a review is frequently an iterative and non-linear undertaking. For example, in one of its more ambitious efforts to identify policy- and practice-relevant topics or knowledge needs, EviEM initiated the so called Knowledge project [41]. It was inspired by the effort to identify 100 highly policy-relevant ecological questions undertaken by Sutherland et al. [42]. The Knowledge project encompassed (1) identification of stakeholders across the environmental sector in Sweden (as described in stage 1); (2) interviews with identified stakeholders regarding their knowledge needs over the next 5 years; (3) collating and clustering of identified knowledge needs (248 in total); (4) identification of experts on subjects within the topic clusters; and (5) a 2-day workshop where subject experts and other stakeholders prioritised their knowledge needs and developed potential review questions. This procedure differed slightly from the previously described 5-stage engagement process as stakeholders, including subject experts, were actively involved in framing and prioritisation of questions before any scoping studies had been conducted. This was done to involve more stakeholders early in the process and to identify a larger number of highly prioritised questions. The project resulted in a list of twelve prioritised topics and four more focused but still not “reviewable” questions, indicating areas where more knowledge is needed for decision-making within Swedish environmental policy and practice. EviEM now uses that list to identify questions suitable for systematic reviews and maps, starting from stage 3 of the engagement process.

Concluding thoughts and lessons learned

The existing CEE guidelines for systematic evidence synthesis in environmental management state that stakeholder engagement is important and should be encouraged [3]. However, there is little guidance on how to identify and engage stakeholders. We have provided an overview of an empirically tested approach to the engagement of stakeholders in early stages of the review process, hoping that this will be useful reading not only for reviewers, but also for stakeholders who plan to participate in the engagement process. In this last section we conclude by providing reflexions and lessons learned from our engagement approach.

To avoid bias stemming from the vested interests of specific groups of stakeholders, it is important to engage with a representative, diverse and well balanced group of stakeholders [43]. EviEM invites participants that represent typical stakeholder groups (e.g. government agencies, industry, NGOs) but also a range of different views within those typical groups (e.g. government agencies with different priorities and targets). However, there can be several obstacles to engage with a representative range of stakeholders. First, it could be difficult to identify all relevant stakeholders. Second, once stakeholders are identified it may be challenging to reach them. Third, it could also be difficult to find a suitable time and place where a representative range of stakeholders can meet. Fourth, regardless of where a meeting is arranged, there is a risk of geographical bias. To minimise that risk, EviEM strives to arrange stakeholder meetings in different parts of Sweden. Another way could be to develop web-based solutions such as e-participation tools and online platforms for knowledge exchange [44]. EviEM has not yet explored such solutions, but on several occasions during the review process, EviEM provides opportunities for stakeholders to engage and send their comments via email.

When prioritising review questions and establishing the scope of reviews, EviEM seeks to conceive systematic reviews and maps that are relevant to a broad range of stakeholders. Such an approach can require significantly more time and resources than synthesis methods used to address narrow review questions with only local applicability [19]. Since parts of the early stakeholder engagement take place before a review is initiated (stages 1–2), it is difficult to calculate the full cost of such efforts per review, but the engagement attributable to a specific review (stages 3–5) requires typically around 2–4 weeks of work by the EviEM review expert managing the review. This effort is fairly small in relation to the entire review process, but if the scope of the review is broadened as a result of the stakeholder engagement, the conduct of the review may become substantially more time-consuming and thus expensive. If a review is commissioned for a specific use by a single user, e.g. a governmental agency, the commissioner may not afford or be interested in expanding the scope of the review beyond their own interest. Therefore, without its current funding model, which offers independent financing of reviews through a research council (see [16]), EviEM would probably not have been able to engage with stakeholders and conduct the reviews using the approach described here. Long-term independent financing clearly improves the conditions for producing syntheses that are relevant to a broad range of stakeholders and provide generalisable results.

Like several other reviewers (e.g. [5]), we have experienced that it can sometimes be challenging to reconcile stakeholders’ desires and expectations with established methods for evidence synthesis (as discussed in “Identification of policy- and practice-relevant topics”and exemplified by the review on PFASs in “ Framing and prioritisation of review questions”). Other authors have argued that a more pragmatic approach, where urgent needs of local stakeholders potentially compromise the comprehensiveness of the review, may be justified [45]. However, the iterative process of prioritisation and scoping employed by EviEM, which involves a continuous dialogue between reviewers, scientists and other stakeholders, usually contributes to scientific rigour while retaining the relevance of reviews to stakeholders.

Early stakeholder engagement can facilitate endorsement of the review, especially when stakeholders feel that they participate actively in review planning and have opportunity to influence the scope and focus of the review [10]. Early engagement also raises awareness among stakeholders of the rigour, transparency and objectivity of systematic reviews and maps. It is therefore reasonable to assume that early engagement of stakeholders may facilitate legitimisation and a wider uptake of review findings in environmental policy and practice [4]. Nevertheless, it is very difficult to measure whether the uptake of reviews would be different if there was no early engagement.

Finally, reviewers can also gain from the stakeholders they are interacting with. Based on our experience as reviewers, early stakeholder engagement helps us to grasp the stakeholders’ sometimes opposing views and potential consequences of the review findings for those concerned. Also, it is very valuable for reviewers to establish a network of stakeholders that will be used in the final stages of the review process, when the results are to be communicated.