Open Access

A framework for stakeholder engagement during systematic reviews and maps in environmental management

  • N. R. Haddaway1Email author,
  • C. Kohl2,
  • N. Rebelo da Silva3,
  • J. Schiemann2,
  • A. Spök4,
  • R. Stewart3,
  • J. B. Sweet5 and
  • R. Wilhelm2
Environmental EvidenceThe official journal of the Collaboration for Environmental Evidence20176:11

DOI: 10.1186/s13750-017-0089-8

Received: 1 August 2016

Accepted: 25 April 2017

Published: 15 May 2017


People have a stake in conservation and environmental management both for their own interests and the sake of the environment itself. Environmental decision-making has changed somewhat in recent decades to account for unintentional impacts on human wellbeing. The involvement of stakeholders in environmental projects has been recognised as critical for ensuring their success and equally for the syntheses of evidence of what works, where, and for whom, providing key benefits and challenges. As a result of increased interest in systematic reviews of complex management issues, there is a need for guidance in best practices for stakeholder engagement. Here, we propose a framework for stakeholder engagement in systematic reviews/systematic maps, highlighting recommendations and advice that are critical for effective, efficient and meaningful engagement of stakeholders. The discussion herein aims to provide a toolbox of stakeholder engagement activities, whilst also recommending approaches from stakeholder engagement research that may prove to be particularly useful for systematic reviews and systematic maps.


Stakeholders Communication Dissemination Methodology Best practice Conflict resolution Stakeholder analysis


Environmental management is a multifaceted subject, influencing humans and the environment alike in a plethora of complex and intricate ways. Conservation and environmental management are of interest to people both because of their own interests and also for the sake of the environment itself. Today, environmental decision-making also accounts for impacts on human wellbeing, for example through the instigation of the ‘at least do no harm’ mandate of the Convention on Biological Diversity [1]. In accordance with the dual recognition of the importance of the environment to human wellbeing, and of human wellbeing in environmental management, the involvement of stakeholders in management projects has been recognised as a critical step in ensuring their success (e.g. [2]). Here, we define stakeholders as being any person or organisation who can affect or may be affected by the planning, conduct, results and communication of a systematic review or map (collectively referred to in the following pages as ‘reviews’), in line with common, broad definitions accepted in the literature (e.g. [3]) (see “Defining stakeholders”, below).1

Stakeholder engagement may provide several key benefits to environmental management research projects (reviewed in [4, 5]), including: improving the evidence base [6]; greater public acceptance [7]; higher likelihood of intervention success [8]; wider communication of findings [9]; and increased likelihood of impact on decision-making [10]. However, engaging stakeholders in research can also be associated with dis-benefits, such as reinforcing power imbalance [11], causing or worsening misunderstandings, and delaying decision-making [12]. However, these negative impacts should not be taken as a reason to avoid stakeholder engagement, but highlight the need for carefully planned, unbiased and balanced engagement.

Stakeholder engagement is associated with a number of challenges that makes its implementation problematic (see Box 1), including: increased demand on time and resources, potential for marginalising or favouring certain groups of stakeholders, biased representation of true stakeholder groups, and tokenistic engagement. Nevertheless, stakeholder engagement has been shown to increase the efficacy of management interventions, particularly where success relates to uptake of activities by practitioners [13].

In the same way as with primary research, reviews can greatly benefit from engaging with stakeholders to ensure that inputs and outputs are of the greatest relevance and reliability to all interested parties. The Guidelines for Systematic Reviews in Environmental Management [14] states that stakeholders play an important role in formulating the review question and advising on the search strategy, and that involving stakeholders at an early stage is of particular importance. Early reviews in conservation and environmental management were, to a large extent, trial cases and focused perhaps more on academic topics (e.g. [15, 16]), or those with restricted groups of identified and engaged stakeholders (i.e. often just the review commissioner) [17, 18]. However recent developments in CEE systematic review and systematic map methodology [19] and an increase in the uptake of systematic review methods in evidence-based conservation and environmental management have resulted in increasing interest in stakeholder engagement throughout review processes.2 As a result there is a need for guidance in best practices for stakeholder engagement.

Here, we formulate a framework for engaging with stakeholders when conducting reviews, highlighting recommendations and advice that may prove useful for effective, efficient and meaningful engagement of stakeholders. We use our experience and a summary of the literature to provide advice for reviewers when deciding which stakeholder engagement activities are priorities, considering which methods are likely to work best in their particular context and, where resources are limited, which methods may be most effective [10]. The existing literature relating to the benefits of stakeholder engagement in reviews is limited, particularly in the field of environmental management where there is a complete knowledge gap. Hence, in addition to being based on an extensive (non-systematic) review of the existing literature on stakeholder engagement generally, this guidance is also based on extensive first-hand experience of reviews, and follows a series of key informant interviews with nine review experts from the fields of environmental management, conservation and social science, all with experience of stakeholder engagement (see Additional file 1 for further details of these interviews). The results of these interviews were used to construct and refine the conceptual models provided herein. This commentary thus goes further than purely reviewing the literature, by complementing the evidence base with experiences of the practicalities of reviews and the required central tenets of systematic review methods.

This document will introduce ideas in stakeholder engagement and provide advice to those designing stakeholder engagement plans for their review. It aims to provide a toolbox of possible stakeholder engagement activities, whilst also recommending approaches from stakeholder engagement research that may prove to be particularly useful for reviews.

Stakeholder engagement and systematic review methods

Stakeholder engagement should reflect systematic review methodology, by being a reliable, transparent process that aims to be as verifiable and objective as possible. Objectivity and repeatability may seem particularly challenging when dealing with groups of people and what may often be strong and variable opinions. However, by maintaining a high level of transparency and clarity, stakeholder engagement can remain a reliable and verifiable process: key tenets of the parallel process of systematic review.

Whilst there is undoubtedly a need for transparency in any stakeholder engagement activities, measures to reduce bias in stakeholder engagement can only be recommended, since appropriate stakeholder engagement methods will be to a great extent context-specific, and available resources for stakeholder engagement may be limited to varying degrees.

Defining stakeholders

Various definitions of stakeholders exist in the literature, with perhaps the most widely cited one being “any group or individual who is affected by or can affect the achievement of an organisation’s objectives” [3]. Reviewers may define the term stakeholder in much the same way (Table 1), although in practice many use the term synonymously with ‘review commissioner’ or ‘end-user’. It may be appropriate, however, to take a broad definition of stakeholders that includes all parties that may affect or be affected by a review. To that extent, we have produced a conceptual model that categorises and separates stakeholders according to three dimensions: who they are, what their roles are, and what actions they may take in relation to the review (Fig. 1). This broad definition includes several key actors that are seldom recognised in definitions, but that we feel should be included to ensure that all affected parties can be given appropriate opportunity for involvement and discussion where suitable, or can be taken into consideration when formulating a stakeholder engagement plan (e.g. research funders). Stakeholders can perform multiple roles within this model. The reader should note that we do not restrict our definitions to ‘end users’, since this definition assumes the reviewers are well aware of (and potentially engaged with) all possible current end users. Our broader definition does not make this assumption.
Table 1

Key informant interviewees’ definitions of the term ‘stakeholder’ with respect to systematic reviews.

Source: unpublished data



“The client. Also experts engaged to do the topic synthesis.”

Novice reviewer

“People who are either affected by the issue or those who may be able to influence the issue: includes local people (e.g. producers), NGOs and governments”

Experienced reviewer

“Anyone with an interest in a particular issue or anyone likely to be affected by an issue or a decision: includes poor people and researchers, research experts (systematic review methodology experts).”

Experienced reviewer

“People that have an interest in the subject matter: includes researchers and experts. Those generating evidence and the end-users of evidence. Also includes subjects of conservation and development projects.”

Experienced reviewer

“A person or representative of an organisation that is affected by an activity that is being reviewed in one way or another: includes scientists.”

Expert reviewer

“Those who have a stake in the question, e.g. policy - makers, academics, educators, NGOs.”

Expert reviewer

“Someone who has a stake in the findingsthe issues have real meaning in their lives; someone affected by the review findings.”

Expert reviewer

“Those in one way or another that use the information from a systematic review: mainly those in decision making (e.g. ministries, agencieson all levels, local, national and international), includes scientists.”

Expert reviewer

Fig. 1

Conceptual model of stakeholders, identified by the actors, their roles and their actions

Guidance 1

Using a broad, encompassing definition of stakeholders can help to ensure that all relevant stakeholders are engaged, particularly minority groups

Why engage with stakeholders?

Stakeholder engagement in reviews is undertaken for several major reasons (see details in Figs. 1, 2): (i) to set the scope and definitions of the review, (ii) to ensure the relevance of the review from a broader society perspective; (iii) to prioritise review questions; (iv) to suggest and locate relevant evidence; (v) to interpret the review findings or set them in context; (vi) to improve the clarity and readability of the review report; (vii) to increase the communication and impact of the review results; and (viii) to endorse the review. Reviewers may have any number of reasons for undertaking stakeholder engagement, but a comprehensive stakeholder engagement strategy will help to ensure that all benefits are felt.
Fig. 2

Model of potential benefits of stakeholder engagement. Models shows direction of benefit with respect to stakeholders (green arrows benefit the review, orange arrows benefit the stakeholders)

We described the major justifications for and benefits of stakeholder engagement briefly above, but some additional specific benefits are worth mentioning. Figure 2 summarises these benefits visually and we give some examples in more detail here. Along with ensuring clarity and readability of the review report, engaging with stakeholders can ensure that processes remain transparent, since additional appraisal of the review process is inherently involved. Furthermore, by identifying, categorising and understanding the characteristics and nature of various stakeholder groups, potential controversies and conflicts during communication of the review results can be anticipated. Along with refining the scope of the review, stakeholders can provide a practical understanding of definitions that may be critical to the review’s inclusion criteria: getting these wrong can significantly reduce the utility of the review’s conclusions [16]. Stakeholders can improve the quality of a review by improving the search strategy, helping to set the balance between specificity and sensitivity, also potentially improving the review’s efficiency. Stakeholders can also improve review quality by providing access to evidence critical to the review; studies or data that are inaccessible, un-indexed, or un-published in academic resources (i.e. grey literature). This may be particularly useful if the evidence base consists of useful data from practitioner-held information, such as consultancy reports, or if non-English language research may be likely. Reviews can be made more relevant through gaining a better understanding of the context in which the findings fit. Similarly, stakeholders can help to tailor communications for the right audience. This is crucial for ensuring that review findings are translated from long, technical review reports into digestible formats that have a high likelihood of being read and integrated into decision making. Stakeholders can help to formulate dissemination media for the review (such as policy briefs, press releases, fact sheets, etc.) that can greatly increase the coverage of dissemination activities and maximise impact. Furthermore, stakeholders may be a useful resource in documenting impact in decision-making, something that has as yet been notoriously difficult to do [20]. Finally, stakeholder engagement may be seen as a key means of promoting evidence-based approaches to decision-making, and may be a useful means of capacity-building. Since any stakeholder engagement related to a review would require an explanation of what systematic review methods involve, this engagement can prove to be highly useful in providing various degrees of training in review methods. Indeed, reviews that engaged with stakeholders throughout the review process may also provide hands-on training that can be particularly effective in building capacity across various communities, including: commissioning high quality reviews using state of the art methods; being able to critique and integrate the results of reviews into decision-making; and, being able to undertake reviews themselves.

Challenges of stakeholder engagement

Stakeholder engagement is not without its challenges. An analysis using systematic review key informant interviews by Cotrell et al. [13] identified a number of key challenges (see Box 1).

Box 1. Key challenges in stakeholder engagement

1. Stakeholder engagement requires additional time and resources

2. Where resources are limited stakeholder engagement must be carefully planned to ensure the results are sufficient

3. Stakeholder engagement may divert resources away from the conduct of the review

4. Reviewers should understand and be transparent about the desired objectives of stakeholder engagement from the outset

5. Tokenism (i.e. activities undertaken for the sake of being seen to undertake them) in stakeholder engagement must be overcome to ensure activities are worthwhile

6. The group of engaged stakeholders may not be balanced and representative

7. Conflicts between stakeholders can arise that must be carefully resolved

8. Reviewers may need training in how to coordinate stakeholder engagement and interact with stakeholders, particularly where conflict may arise, or an expert in conflict management may be needed

9. Inputs from multiple different stakeholders can sometimes contradict and processes must be in place to develop and deal with the situation and any compromises that may be necessary

10. Stakeholders must be briefed in systematic review methods carefully to ensure full understanding and appreciation of the possible inputs they can have, whilst avoiding confusion

11. Specific criteria may be needed to aid stakeholders in prioritisation of review questions

12. Difficulties in maintaining continued engagement throughout the review to maintain interest and involvement

13. Reviewers must bear in mind stakeholders’ concerns about confidentiality when writing up their findings and discussing stakeholder inputs with other parties

14. Contributions to the review by all involved, including stakeholders, need to be taken into account when deciding on authorship, drafting acknowledgements and attributing credit

15. Reviewers must ensure there is no undue influence on the review as a result of stakeholder engagement (and must define what constitutes undue influence; possibly defined as a significant change to the scope or content of the review as a result of the opinion of one or more stakeholders not broadly accepted by the stakeholder group)

One significant challenge with stakeholder engagement is that of explaining systematic reviews to those with no previous experience of the methodology (Box 1, point 10). We recognise here that our broad definition of stakeholders includes review experts, but focus here on the common groups of stakeholders (including researchers) who are not aware of systematic review methods. Systematic review methods in environmental management are relatively novel, with a small minority of the research community aware of precisely what is involved in a formal CEE review. The concepts and processes involved in a CEE review are the subject of extensive documentation and training ([14];, and an understanding of these processes requires a solid grasp of the way in which academia and science research publishing work. There is thus a significant challenge in explaining systematic reviews to stakeholders sufficiently that their involvement is meaningful without overloading them with jargon and complicated novel concepts. Organisations working closely with stakeholders and reviews have approached this challenge in slightly different ways, but minimising the amount of new information regarding methodology may be useful initially, whilst also allowing stakeholders to gain further awareness using online information and tutorials at their own speed. Workshops and meetings may be started with short presentations that provide a brief overview of systematic review methodology. There may be a need for a ‘soft landing’ when it comes to explanations of systematic review; perhaps through the use of infographics or explanatory booklets in lay terms.

Another challenge is the importance of clearly stating the objectives of stakeholder engagement from the outset (Box 1, point 4). Providing stakeholders with examples of the ways in which they can be involved and the types of information and inputs they can supply can be critical to ensuring that resources are used efficiently and inputs are meaningful. For example, it may be useful to give examples of the types of comments on a protocol that would be useful along with the types of comments that might not. Equally it is important to outline exactly what stakeholders can expect from stakeholder engagement; being clear about what they can influence and what they cannot. For example, they may be able to contribute to defining the inclusion criteria but they cannot influence decisions about which studies are included. Procedures for dealing with conflict and contradictions should also be specified from the outset. For example, experienced or trained mediators or facilitators may be used in physical meetings (Box 1, point 8), and where conflicts cannot be resolved, lead reviewers may be given the final say in the approach used in the review.

Whilst some people may fear that stakeholder engagement can derail a review, such an outcome should not be possible (unless the derailment is warranted, for example if the review question or outcomes under consideration are deemed inappropriate). Stakeholder engagement should help to direct a review by providing advice and experience to the review team: it should not be able to unduly influence it (i.e. their influence should not significantly alter review methodology unless agreed by all/the majority of stakeholders and the review team of subject and methodology experts (reviewed by Oliver et al. [21]). The transparent, objective, and verifiable methods used for the review running parallel to stakeholder engagement should remain robust to any potential stakeholder bias or undue stakeholder influence. However, stakeholder engagement could potentially reduce the efficiency of review activities, where significant resources are diverted to this task, reducing those available for the conduct of the review.

Stakeholder engagement in practice

Stakeholder analysis and balance

Stakeholder analysis is the process by which key actors in a system are identified, categorised and understood [22]. It has been used extensively in dispute resolution and conflict management [22, 23], but is likely an implicit aspect of most stakeholder engagement activities, despite perhaps not being recognised as such. Stakeholder analysis is undertaken for a range of reasons, including: (i) ensuring balance in stakeholder groups; (ii) prioritising certain groups of stakeholders over others where resources are limited; (iii) identification and investigation of possible conflicts between stakeholders; (iv) tailoring contact to specific types of stakeholder; and (v) phasing contact with stakeholders through a project according to their relevant utility to and benefit from the research. Whilst always a useful exercise, stakeholder analysis may not always be necessary; for example, where the stakeholders for a given subject are already well known.

Stakeholder analysis is typically done using a range of methods that relate to the categorisation of stakeholders [24]. The majority of these methods are ‘top-down’: they involve categorisation of stakeholders by a third party, usually a reviewer [25]. There are also ‘bottom-up’ methods, however, that involve (amongst other things) categorisation of the various stakeholders by other stakeholders [25]. This latter method can be particularly resource intensive, however, but can identify intricacies that would otherwise be missed by reviewers less familiar with the existing relationships amongst stakeholders.

The most common means of stakeholder analysis is via interest-influence matrices [26]. These matrices classify stakeholders along two dimensions; interest in the subject in hand and influence in the processes involved. Such matrices allow the identification of key stakeholders that should be targeted (e.g. high interest, high influence), or modified (e.g. low interest, high influence) by attempting to increase their level of interest. Other dimensions can be plotted on such matrices, for example, amount of available evidence and necessary engagement effort. Other means of categorisation described in the stakeholder analysis literature include semi-structured interviews, snowballing, social network analysis, knowledge mapping, etc. (reviewed by Brugha and Varvasovszky [27]). Further details on these methods should be sought from the extensive methodological literature.

Generally speaking, ‘balance’ in stakeholder engagement may be understood as the representation of all main interests, views and opinions [28], but its application in practice is challenging. What stakeholder engagement balance should not be aiming for is the strict proportional, quantitative representation of stakeholder groups present in society. There are many reasons for avoiding quantitative balance in stakeholder engagement. Firstly, activities such as workshops are unlikely to be able to cater for and attract all relevant stakeholder groups, so ensuring that one representative from every group is present is likely to be an intangible aim. Furthermore, some individuals may represent larger stakeholder groups, whilst other individuals represent only themselves. Such group representatives, however, may have collated the views and speak on behalf of a large number of individual stakeholders. Accordingly, quantitative assessments may ignore the underlying numbers. The key aim with ensuring balance is to allow all major types of stakeholder to be given the opportunity to provide input. For example, where a review focuses on the environmental impacts of ‘fracking’, a balanced stakeholder pool might consist of citizens, policy-makers, conservation practitioners, representatives from the fossil fuel industry, land-owners, representatives from regional water boards, etc. Balance is most evident when it is absent: for example, through the notable absence of fossil fuel industry representatives. In this way, stakeholder analysis can help to identify where balance is lost by categorising stakeholders and predicting which groups should be present.

Often, resources for stakeholder engagement are limited (Box 1, points 1–3), and key stakeholders must be prioritised and contacted in preference to others. This pragmatic approach should be undertaken carefully to ensure that balance is still maintained wherever possible, and that the engaged stakeholders provide knowledge and opinions that are representative of or accepted by the stakeholder community at large. Basing a review on definitions that are not broadly accepted, for example, can drastically reduce the worth of a review [29]. Similarly, using stakeholder analysis to phase contact with stakeholders may be a useful way of dealing with limited resources with maximum efficiency. Equally, the phased inclusion of stakeholders should be based on sound analysis of the risks of excluding certain groups from specific processes. It should also be noted that systematic reviews are lengthy processes, and long gaps between contact for those phased stakeholders that are involved more than once can lead to significant stakeholder attrition (drop out) if contact is not suitably maintained. Stakeholder analysis can also be useful in identifying groups of stakeholders that might benefit from different forms of contact. Certain groups may interact more if contacted in a specific way or at a specific time; such as teachers being more responsive by telephone after school hours. Other groups may require different wording in emails, for example, if their level of understanding of academic literature and systematic review methods is expected to be higher. Stakeholder analysis can also be a vital means of early identification of potential conflicts between different stakeholder groups. Such awareness can be crucial for ensuring that stakeholder engagement activities run smoothly, for example by interacting with conflicting groups at different meetings rather than assembling them in one room. Finally, stakeholder analysis can also help to identify potential bias that may reduce stakeholder engagement balance. Sources of bias in stakeholder engagement are discussed in further detail below.

Guidance 2

Stakeholder analysis can help to identify and sort stakeholders according to useful categories related to the type of benefits they might provide to the review or receive from it. Stakeholder analysis can help ensure balance, prioritise limited stakeholder engagement resources, help to identify or anticipate potential conflicts, and assist in tailoring and phasing contact with stakeholders, particularly if reviewers are less familiar with or uncertain of the stakeholder community linked to a certain review question

Stakeholders and review stages

Prioritising and phasing stakeholder engagement should be based on a sound understanding of the major stages in a review, and a clear appreciation of the types of roles and actions that stakeholders can perform. These actions are summarised in Fig. 1. Tables 2 and 3 display the review stages of the major activities within a review, and the relationship between stakeholder actions, review stages and direction of information flow, from question formulation through to communication and integration of results into decision-making. Engagement activities can be focused towards different groups of stakeholders depending on the actions they are believed to be able to perform. Care must be taken to avoid surprises relating to actions performed by stakeholders that were not identified in advance, since this information may come too late and may risk full endorsement and acceptance of the review.
Table 2

Systematic review processes and stages



Question formulation






Article screening


Data extraction


Critical appraisal




Final review






Table 3

Stakeholder actions, systematic review stages and directions of information flow


Review stage(s)

Direction of action

Share own experiences


Review ←

Share articles

Early, mid

Review ←

Endorse the review

Early, mid, Final

Review ← → stakeholders

Undertake the review

Early, mid, final

Review ←

Share missing/supplementary information for specific studies


Review ←

Provide context

Early, final

Review ←

Provide funding and/or in-kind contributions

Early, final (communication)

Review ←

Represent an organisation/group

Early, final

Review ←

Set review standards

Early, final

→ Stakeholders

Share knowledge

Early, final

→ Stakeholders

Facilitate access to the review


→ Stakeholders

Read the review

Final, post

→ Stakeholders

Share the review

Final, post

→ Stakeholders

Integrate review findings into decisions


→ Stakeholders

Table produced during a workshop to analyse the results of key informant interviews

The need for acknowledgement

As stated above, stakeholder engagement activities should be transparently documented throughout the process. In addition, the role of stakeholder engagement in a review should be clearly stated in some form of acknowledgement. Stakeholders may have contributed considerable time and resources to a review, and acknowledgement for their efforts is not only fair and important for transparency reasons but also a sensible courtesy. Such acknowledgement can take a variety of forms, including: (i) co-authorship of review protocols, reports and resulting publications (although caution must be taken to ensure the review maintains sufficient independence and acceptance by the broader stakeholder group); (ii) documentation in methods text within reports; and (iii) mentioning within acknowledgement sections of dissemination media, including formal publications. Care should be taken if stakeholders are to be mentioned specifically, and permission should always be sought before providing any names of individuals or organisations. Care should also be taken where conflicts may arise through publishing individual names of stakeholders, and referring to generic groups or categories of stakeholders may be preferable if anonymity is desired.

Guidance 3

Acknowledgement of stakeholder contributions should be carefully considered in order to ensure transparency and to thank significant contributors. Permission should be sought before naming specific stakeholders in order not to violate privacy and/or not to be harmful to them in their professional capacity

Sources of bias in stakeholder engagement

A variety of biases can find their way into stakeholder engagement that can reduce the efficacy of the engagement process. Here, we refer to biases in the stakeholder engagement process itself, not bias held by any individual stakeholder. Reviewers should not attempt to mitigate individual’s biases, but rather attempt to engage with stakeholders in an unbiased way. In the following section we discuss those biases in turn, and provide details of how the bias can be avoided or mitigated. Since it is the reviewers who hold the power in this situation, it is they who must be responsible for identifying, understanding and (where possible) mitigating these biases. We discuss these biases across three key aspects of stakeholder engagement; stakeholder selection, stakeholder response and stakeholder attrition. Each bias is summarised along with suggested mitigation measures in Table 4.
Table 4

Potential biases that may arise in outputs of stakeholder engagement





Stakeholder selection

Identification bias

Purposeful selection of stakeholders using personal/organisational knowledge or unsystematic searches may result in a biased and unbalanced group of stakeholders

Use a combination of selection methods

Network bias

Asking others to suggest potential stakeholders may result in a biased and unbalanced group of stakeholders

Use multiple starting points (suggestees) from a range of backgrounds

Awareness bias

Announcing an open call for stakeholder engagement may target a biased and unbalanced group of stakeholders

Advertise the open call using a range of different channels, using stakeholder analysis to identify stakeholders that may require specific forms of contact

Self-promotion bias

Systematically searching for potential stakeholders may select only those with an online presence, producing a biased or unbalanced group of stakeholders

Use a combination of selection methods

Stakeholder response

Awareness bias

Announcing an open call for stakeholder engagement may target a biased and unbalanced group of stakeholders

Advertise the open call using a range of different channels, using stakeholder analysis to identify stakeholders that may require specific forms of contact

Access/technology bias

Stakeholders may not have the ability to respond to invitations, producing a biased, unbalanced group of stakeholders

Provide multiple modes of engagement that do not rely purely on one technology/format

Intimidation bias

Stakeholders may be less likely to respond if they feel their views are unlikely to be heard over the views of the majority

Provide support to minority stakeholders by tailoring contact and ensuring that views will be heard in initial invitations

Faith bias

Stakeholders may not engage if they believe that their views will not be heard due to failures on the part of the reviewers

Undertake stakeholder analysis to help identify and categorise potential conflicts. Ensure openness and contactability to support and facilitate response from less vocal and minority stakeholder groups

Apathy bias

Stakeholders may not respond if they feel others will perform their role for them

Encourage stakeholders to engage by explaining that all views are valid and important, and stress the need for a comprehensive, balanced group of stakeholders

On-going engagement

Commitment bias

Stakeholders may not be able to commit to involvement along the full extent of the systematic review process, causing attrition over time and leaving a biased, unbalanced group of stakeholders

Phase contact with certain stakeholders according to their likely involvement

Timescale bias

Long timescales involved with systematic reviews may mean that attrition occurs over time as stakeholders change roles, in turn leaving a biased, unbalanced group of stakeholders

Attempt to engage with multiple stakeholders from each organisation to ensure some contacts remain

Resource bias

Stakeholders’ resources may be too limited to allow full engagement throughout the systematic review process, leaving a biased, unbalanced group of stakeholders

Phase contact with certain stakeholders according to their likely involvement. Minimise necessary resources needed for engagement, for example by reducing unnecessary reading

Access/technology bias

Stakeholders may not have the ability to respond to invitations or on-going engagement, resulting in attrition and leaving a biased, unbalanced group of stakeholders

Provide multiple modes of engagement that do not rely purely on one technology/format

Intimidation bias

Stakeholders may be less likely to respond if they feel their views are unlikely to be heard over the views of the majority

Provide support to minority stakeholders by tailoring contact and ensuring that views will be heard in initial invitations

Faith bias

Stakeholders may not engage if they believe that their views will not be heard due to failures on the part of the reviewers

Undertake stakeholder analysis to help identify and categorise potential conflicts. Ensure openness and contactability to support and facilitate response from less vocal and minority stakeholder groups

Apathy bias

Stakeholders may not respond if they feel others will perform their role for them

Encourage stakeholders to engage by explaining that all views are valid and important, and stress the need for a comprehensive, balanced group of stakeholders

Stages include; stakeholder selection, stakeholder response and on-going engagement along with suggestions for mitigation

Stakeholder selection

Bias in stakeholder selection arises through the methods used to identify and invite stakeholders. There are four ways in which stakeholders can be invited to engage; purposive selection, ‘snowballing’, open calls, or systematic selection (Fig. 3). Bias can arise with any mechanism used to identify stakeholders. Purposive selection involves the identification and invitation of selected, often well-known stakeholders. This process potentially results in a biased sample of stakeholders and risks excluding minorities (identification bias [30]). This method is often preferred, however, since reviewers are usually familiar with the targeted stakeholders, meaning that there may be higher chances of positive responses, and more manageable numbers of engaged stakeholder groups. ‘Snowballing’ describes the use of an initial list of stakeholders who are then asked to propose further stakeholders, continuing on in a rapidly expanding manner. Snowballing can also result in identification bias, and can further compound the risks of ignoring minorities by repeating the same bias across multiple stakeholders (network bias [31]). Multiple iterations of snowballing with several different starting points (perhaps using key stakeholder groups or known minority stakeholders) can reduce network bias. Open calls for stakeholder engagement can generate much greater numbers of interested stakeholders, with a potentially wider diversity by avoiding identification and network bias. However, open calls risk missing those with no access to the advertisement (we propose the term awareness bias). It may also produce an unmanageable number of interested stakeholders, and minority views may be swamped if all parties are engaged yet reviewers’ resources are limited. Systematic identification of stakeholders mirrors the systematic approaches used to identify evidence within a review, and involves a search for potential stakeholders, for example the Poverty and Conservation organisations database (; an IIED project). Systematic approaches are inherently less likely to suffer from bias and they use a verifiable, justified methodology. However, large numbers of stakeholders may be identified, and use of the method online risks missing those without an online presence (self-promotion bias [32]). Systematic searches may be the least biased method for identifying stakeholders, but using several methods together is likely to result in the least overall bias and unbalance in the stakeholder pool.
Fig. 3

Methods of stakeholder invitation with explanations (italics) and their relative advantages (red text) and disadvantages (green text)

Stakeholder responses to invitations

Once stakeholders have been identified, invitations to engage should be sent out. Stakeholder response to invitations is another point at which bias can manifest itself. As described above, stakeholders can only respond to open calls if they are aware of them (awareness bias). In addition, the ability to respond to invitations requires that stakeholders have the ability to contribute (e.g. access to an email account), something that may prove challenging in certain situations (we propose the term access/technology bias). Stakeholders may not be able to attend physical meetings due to limited time or money (resource bias [33]). Minority stakeholder groups may not respond if they feel their views are unlikely to be heard over the views of the majority (intimidation bias [34]). Similarly, stakeholders may not engage if they believe that their views will not be heard due to failures on the part of the reviewers or the review methodology (we propose the term faith bias). Finally, some stakeholders may not respond if they feel others will perform their role for them (we propose the term apathy bias).

These biases can be mitigated in a range of ways, including: (i) using multiple advertisement channels for open calls to maximise the target audience; (ii) providing multiple modes of response for those who may not have access to the internet; (iii) providing support to minority stakeholders by tailoring contact and ensuring that views will be heard in initial invitations; (iv) undertaking stakeholder analysis to identify and avoid potential conflicts between stakeholders; (v) offering financial support for attending meetings to those with limited budgets; and (vi) ensuring openness and contactability to support and facilitate response from less vocal and minority stakeholder groups.

Stakeholder attrition in on-going engagement

Stakeholder attrition (drop out) can result in a change in the proportion and balance of stakeholders throughout the on-going engagement and review processes. Differing abilities to commit to the long-term nature of stakeholder engagement throughout a full review can result in loss of important stakeholders (we propose the term commitment bias). Another effect of the extensive nature of systematic review timescales is the loss of stakeholders through changes in individual job roles over time (we propose the term timescale bias). As mentioned above, stakeholders may drop out of stakeholder engagement if resources are limited (resource bias). Similarly, access/technology, intimidation bias, apathy bias, and faith bias may continue to be a problem throughout continued engagement. It is important to note that attrition will clearly be more significant as time goes on, meaning that the type and degree of input from stakeholders engaged throughout the review process is likely to diminish in latter stages of the review.

These biases may be mitigated in several key ways. Limitations in the ability of stakeholders to commit resources to engagement can be combated by phased contact with resource-limited stakeholders to ensure that they are contacted at the most appropriate stages and not overtasked. This is particularly important where certain stakeholders are needed for structuring communication activities. Providing stakeholders with multiple modes of interaction (for example workshops, email, face-to-face interviews, etc.) can mitigate access/technology bias and resource bias. Alternatively, resources to aid travel to physical meetings can be provided to stakeholders. Engaging with multiple stakeholders from key organisations can ensure that organisations are not lost if staff changes occur. Finally, providing stakeholders with an encouraging, supportive environment and ensuring openness and contactability (as described above) can reduce intimidation, apathy and faith biases.

Guidance 4

Potential for bias to occur in stakeholder engagement can be mitigated by using a carefully planned, systematic approach to stakeholder engagement. Employing a combination of methods to identify stakeholders, using multiple start points for iterative identification techniques (such as snowballing), and systematically searching for stakeholders can mitigate biases during stakeholder identification. Contacting stakeholders and allowing them to respond via multiple channels (e.g. in person, by post and via email) can mitigate biases during stakeholder responses. Avoiding overtasking by phasing contact, providing multiple methods of interaction, including redundancy within stakeholder organisations through multiple contacts, and providing a supportive, encouraging environment can mitigate biases during on-going engagement. Finally, careful planning can identify potential bias, for example using stakeholder analysis, and allow for attempts to be made to mitigate bias

Measuring the Success and Impact of Stakeholder Engagement

A further specific challenge relates to how we measure ‘success’ in stakeholder engagement. Reviewers should consider the original objectives of the stakeholder engagement process for their specific review, commonly to: assist in defining a scope that is broadly relevant to a wide range of stakeholders; provide additional evidence where available; endorse the methods used in the review; and, contribute to communication of the review findings.

Many reviewers would define success as improving the quality, communication and impact of a review, but success should also include meeting the objectives of the stakeholder engagement process itself. Issues that are related to success include: (i) a feeling of inclusion and opportunities to be heard by all relevant stakeholders; (ii) acceptance of systematic review methods as a reliable means of summarising scientific evidence on a topic; (iii) faith in the review findings as a robust synthesis of the evidence; (iv) trust in the review team, which can be a particular problem when tackling controversial issues. These latter definitions of success are likely to be affected by stakeholders’ beliefs of ‘what counts as evidence’, and care should be taken to ensure they are aware that systematic review is only one tool for summarising evidence, albeit often seen as the ‘gold standard’ method [35]. Closely related to measuring success, impact should be documented in stakeholder engagement activities in order to show how stakeholders were able to input to the formulation and undertaking of the review. This is also closely related to acknowledgement and transparency. A further benefit from monitoring impact, however, is that future stakeholder engagement activities may benefit from critical assessment of procedures that resulted in optimum impact.

Guidance 5

Reviewers should document any impact that stakeholder engagement may have on the review. Reviewers may also wish to formally (internally or externally) attempt to critically assess success or impact to help improve future stakeholder engagement activities. Making results of this assessment open access would also benefit the wider systematic review and stakeholder engagement research community

Framework for stakeholder engagement

As we have discussed, stakeholder engagement should be undertaken in a transparent manner, attempting to minimise bias where possible. The framework outlined in Box 2 may act as a checklist for those planning engagement activities. It does not form a rigid set of requirements: rather, it is comprised of a suite of potentially suitable methods for maximising balance and minimising bias throughout engagement. The framework provides a guide for processes that may be useful and beneficial in ongoing communications with stakeholders.


Stakeholder engagement can be a time-consuming and resource-demanding process, but it is widely felt by systematic review experts that it should form an integral part of all systematic reviews to some degree or other, particularly at early stages of the review. If planned carefully, stakeholder engagement can be a resource-efficient process that provides a variety of tangible benefits to the scope, processes, quality and acceptance of outputs of a review. But more than this, stakeholder engagement can be vital for ensuring that review outputs have the greatest relevance and impact to the stakeholders that will be the end-users of and/or affected by the review. Reviewers should be aware of potential pitfalls of stakeholder engagement, avoiding bias and striving for balance. By following the suggestions and recommendations in this guidance, we hope that reviewers can increase the effectiveness and efficiency of the planning, conduct and reporting of stakeholder engagement activities during systematic reviews and maps.

Box 2. Approaches and tools for stakeholder engagement in systematic review/systematic map considering measures to ensure balance and mitigate bias

1. Identification of stakeholders

 a. Stakeholder analysis

  i. Check for balance

  ii. Prioritise certain stakeholders

  iii. Tailor engagement activities

  iv. Phase engagement

  v. Identify potential conflict/bias and plan for mitigation

 b. Selection process

  i. Purposive selection

  ii. Snowballing

  iii. Open call

  iv. Systematic approach

2. Initial invitation

 a. Invitation type, i.e. open call/advertisement versus closed invitation (selected stakeholders only)

 b. Invitation format (e.g. email/telephone/conference presentation) and wording

 c. Tailor invitation to specific stakeholders/stakeholder groups

 d. Clarify purpose and format of stakeholder engagement

3. Initial engagement

 a. Format

  i. Group meeting/workshop

  ii. 1-on-1

  iii. Remote (email, online or post)

 b. Plan for dealing with conflict

  i. Involve experienced mediator/facilitator

  ii. Modify engagement format to minimise conflict

  iii.Plan for dealing with unresolvable conflict, i.e. where compromise would impact the review

4. Explanation of subject-specific terminology

 a. Level of explanation of systematic review methodology

  i. Brief explanation in contact

  ii. Links to external sources of additional information

  iii. Full explanation (tailored to specific stakeholders)

  iv. Explanation through presentation in workshops/meetings

 b. Agreement on contentious definitions and terminology

 c. Avoid jargon

5. Maintaining interest throughout the process

 a. Level of on-going communication with stakeholders

  i. Regular contact to avoid lack-of-interest attrition

  ii. Warn stakeholders of potential reduced contact during review activities

6. Documenting stakeholder inputs to the review

 a. Include explanation of inputs from stakeholders (specified, where suitable) that affected the review scope/methodology in the protocol and final review

7. Dissemination and communication

 a. Dissemination media format

  i. Review only

  ii. Review plus press release

  iii. Multiple media tailored to specific stakeholder groups

  iv. Advertising of published media (e.g. blogs, social media)

 b. Active versus passive dissemination

  i. Review outputs (dissemination media) published online

  ii. Dissemination media sent to stakeholders

   1. Subgroup of active stakeholders

   2. All identified stakeholders

8. Acknowledging stakeholder contributions

 a. Acknowledge all engaged stakeholders

 b. Obtain informed consent before naming specific stakeholders

 c. Describe planned/completed stakeholder engagement activities in the protocol and final review

9. Eliciting Feedback on stakeholder engagement activities

 a. Request feedback on perceived success of stakeholder engagement process from stakeholders (i.e. opinions and comments)

 b. Use feedback to assess success of engagement process

  i. Define stakeholder engagement success as

   1. Stakeholder feeling of inclusion

   2. Stakeholder opinions taken into consideration

   3. Stakeholder endorsement of the review

10. Critical self-assessment

 a. Evaluate stakeholder engagement processes internally

 b. Evaluate stakeholder engagement processes externally by independent body

 c. Publish findings of evaluation

 d. Alter processes in the future where necessary


The literature cited in “Background” refering to the term ‘stakeholders’ uses a range of different definitions for who those stakeholders might be, sometimes meaning direct users of research outputs, such as policy decision makers and practitioners (e.g. land managers in the field of environmental management), and sometimes meaning those directly affected by decisions (e.g. patients in the field of medicine).


A search of Web of Science Core Collections on 18th April 2017 using the term "stakeholder engagement" AND "systematic review" as a topic word search yielded an exponentially increasing number of publications.



Authors’ contributions

NRH undertook key informant interviews. NRH and NRdS undertook thematic analysis of survey results. NRH, NRdS and RS formed a working group to discuss thematic analysis results and prepare conceptual models. NRH drafted the manuscript and all authors contributed to the final manuscript. All authors read and approved the final manuscript.


The authors wish to thank the nine key informants involved in the survey for this manuscript. They also thank all authors’ affiliation institutions for allowing time and resources to collaborate on this manuscript.

Competing interests

The authors declare that they have no competing interests.

Availability of data and materials

Key informant responses have not been published for reasons of confidentiality. All other data are published in this manuscript or as supplementary information.

Consent for publication

All participants in the survey provided consent to participate under the provision of anonymity.

Ethics approval and consent to participate

Ethics approval for this study was granted by the School of Earth, Natural Resources and Geography (SENRGy), Bangor University.


This work has been produced in the absence of dedicated funding.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

Mistra EviEM, Stockholm Environment Institute
Federal Research Centre for Cultivated Plants, Julius Kühn-Institut (JKI)
CEE Jo’burg, Centre for Anthropological Research (CfAR), University of Johannesburg
Alpen-Adria Universität Klagenfurt|Wien Graz and IFZ-Inter-University Research Centre for Technology, Work and Culture
JT Environmental Consultants (JTEC)


  1. Smith J, Mapendembe A, Vega A, Hernandez Morcillo M, Walpole M, Herkenrath P. Linking the thematic programmes of work of the convention on biological diversity (CBD) to poverty reduction. Biodiversity for development: new approaches for national biodiversity strategies. Montreal: CBD Secretariat; 2010.Google Scholar
  2. Enters T, Young A, Boshier D, Boyle T. Rethinking stakeholder involvement in biodiversity conservation projects. In: Forest conservation genetics: principles and practice. 2000. Wallingford, UK: CABI Publishing; p. 263–73.
  3. Freeman R. Stakeholder management: a strategic approach. New York: Pitman; 1984.Google Scholar
  4. Reed MS. Stakeholder participation for environmental management: a literature review. Biol Conserv. 2008;141(10):2417–31.View ArticleGoogle Scholar
  5. Sterling EJ, Betley E, Sigouin A, Gomez A, Toomey A, Cullman G, Malone C, Pekor A, Arengo F, Blair M, Filardi C, Landrigan K, Porzecanski AL. Assessing the evidence for stakeholder engagement in biodiversity conservation. Biol Conserv. 2017;209:159–71.View ArticleGoogle Scholar
  6. Reed MS, Dougill AJ, Baker TR. Participatory indicator development: what can ecologists and local communities learn from each other. Ecol Appl. 2008;18(5):1253–69.View ArticleGoogle Scholar
  7. Richards C, Carter C, Sherlock K. Practical approaches to participation, Citeseer. 2004.
  8. Dougill A, Fraser E, Holden J, Hubacek K, Prell C, Reed M, Stagl S, Stringer L. Learning from doing participatory rural research: lessons from the Peak District National Park. J Agric Econ. 2006;57(2):259–75.View ArticleGoogle Scholar
  9. Reed M, Dougill A. Linking degradation assessment to sustainable land management: a decision support system for Kalahari pastoralists. J Arid Environ. 2010;74(1):149–55.View ArticleGoogle Scholar
  10. Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, Tunis SR. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. 2012.
  11. Nelson N, Wright S. Power and participatory development: theory and practice. Athens, Greece: ITDG Publishing; 1995.View ArticleGoogle Scholar
  12. Vedwan N, Ahmad S, Miralles-Wilhelm F, Broad K, Letson D, Podesta G. Institutional evolution in Lake Okeechobee management in Florida: characteristics, impacts, and limitations. Water Resour Manag. 2008;22(6):699–718.View ArticleGoogle Scholar
  13. Cottrell E, Whitlock E, Kato E, Uhl S, Belinson S, Chang C, Hoomans T, Meltzer D, Noorani H, Robinson K. Defining the benefits of stakeholder engagement in systematic reviews. 2014.
  14. CEE. Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2., The Collaboration for Environmental Evidence, 4.2. 2013.
  15. Davies ZG, Pullin AS. Do hedgerow corridors increase the population viability of woodland species? Systematic Review No. 8 Part A. Technical report. Birmingham: Centre for Evidence-Based Conservation; 2006.
  16. Stewart GB, Coles CF, Pullin AS. Applying evidence-based practice in conservation management: lessons from the first systematic review and dissemination projects. Biol Conserv. 2005;126(2):270–8.View ArticleGoogle Scholar
  17. Martínez-Abraín A, Oro D, Jiménez J, Stewart G, Pullin A. What are the impacts of human recreational activity on the distribution, nest-occupancy rates and reproductive success of breeding raptors? CEE review 07-003 (SR27). Collaboration for Environmental Evidence; 2008.
  18. Pullin AS, Bangpan M, Dalrymple S, Dickson K, Haddaway NR, Healey JR, Hauari H, Hockley N, Jones JP, Knight T. Human well-being impacts of terrestrial protected areas. Environ Evid. 2013;2(1):19.View ArticleGoogle Scholar
  19. James KL, Randall NP, Haddaway NR. A methodology for systematic mapping in environmental sciences. Environ Evid. 2016;5(1):7.View ArticleGoogle Scholar
  20. Bunn F. The impact of systematic reviews on health care policy in England. 2010.
  21. Oliver S, Liabo K, Stewart R, Rees R. Public involvement in research: making sense of the diversity. J Health Serv Res Policy. 2015;20(1):45–51.View ArticleGoogle Scholar
  22. Ramirez R. Stakeholder analysis and conflict management. In: Cultivating peace: conflict and collaboration in natural resource management. International Development Research Centre (Canada), World Bank Institute; 1999. p. 101–26.
  23. Swiderska K. Implementing the rio conventions: implications for the south. iied Opinion, special issue. London: IIED; 2002.
  24. Byrson J. What to do when stakeholders matter. Public Manag Rev. 2004;6(1):21–53.View ArticleGoogle Scholar
  25. Grimble R, Chan MK. Stakeholder analysis for natural resource management in developing countries. In: Natural resources forum. New York: Wiley; 1995.
  26. Reed MS, Graves A, Dandy N, Posthumus H, Hubacek K, Morris J, Prell C, Quinn CH, Stringer LC. Who’s in and why? A typology of stakeholder analysis methods for natural resource management. J Environ Manag. 2009;90(5):1933–49.View ArticleGoogle Scholar
  27. Brugha R, Varvasovszky Z. Stakeholder analysis: a review. Health Policy Planning. 2000;15(3):239–46.View ArticleGoogle Scholar
  28. Ihugba BU, Osuji OK. Corporate citizenship and stakeholder engagement: maintaining an equitable power balance. 2011.
  29. Stewart GB, Pullin AS, Coles CF. Effects of wind turbines on bird abundance. CEE review 04-002 (SR4). Collaboration for Environmental Evidence; 2005.
  30. Leventon J, Fleskens L, Claringbould H, Schwilch G, Hessel R. An applied methodology for stakeholder identification in transdisciplinary research. Sustain Sci. 2016;11(5):763–75.View ArticleGoogle Scholar
  31. Wellen R, Toronto O. Open access scholarly publishing and the problem of networks and intermediaries in the academic commons. Univ.-Bibliothek; 2005.
  32. Sears C. Revising the system of corporate tax loss transfers in Canada, University of New Brunswick, Faculty of Business Administration. 2013.
  33. Berggren HM. Institutional context and reduction of the resource bias in political sophistication. Political Res Q. 2001;54(3):531–52.View ArticleGoogle Scholar
  34. Cramer CK, Klasser GD, Epstein JB, Sheps SB. The Delphi process in dental research. J Evid Based Dent Pract. 2008;8(4):211–20.View ArticleGoogle Scholar
  35. Pullin AS, Knight TM. Effectiveness in conservation practice: pointers from medicine and public health. Conserv Biol. 2001;15(1):50–4.View ArticleGoogle Scholar


© The Author(s) 2017