Engaging environmental policy-makers with systematic reviews: challenges, solutions and lessons learned

Abstract

The creation and accumulation of robust bodies of knowledge, along with their dissemination, utilisation and integration in decision support are key to improving the use of evidence in decision-making. Systematic reviews (SRs), through their emphasis on transparency, replicability and rigour, offer numerous benefits throughout the policy-making cycle and for improving the use of evidence in environmental policy-making. As a result there have been numerous calls to increase the use of SRs in environmental policy-making. This commentary paper introduces the challenges of engaging policy-makers with SRs and, using experiences of producing SRs with Government Departments and Agencies within the UK and Europe, identifies possible solutions and shares our lessons learned. It highlights that co-production can help to overcome a number of challenges by ensuring that review questions are policy-relevant, that the context of the review is taken into consideration and that review’s findings are communicated so that they are recognised and used in policy decision-making processes. Additionally, a pragmatic approach to the review’s methodology may be required to respond to policy-making requirements. Here, a risk-based approach can communicate the trade-offs between the rigour and timeliness of the review. Ensuring that systematic approaches are upheld at all times can help address impartiality concerns and can develop skills in both reviewers and policy-makers to increase awareness of systematic methods, leading to changes in practice and culture within decision-making organisations and the promotion of evidence informed policy development and decisions.

Introduction

The creation and accumulation of robust bodies of knowledge, along with their dissemination, utilisation and integration in decision support are key to improving the use of evidence in decision-making [1]. Systematic reviews (SRs), through emphasis on objectivity, rigour, replicability and transparency offer numerous benefits in the policy-making process and will be particularly important where outcomes may influence high-impact decisions that affects those who have either a direct or indirect a stake, known hereafter as stakeholders [2], or in areas that may be contested and controversial [3]. As environmental management is characterised by complexity, with multiple stakeholders and uncertain ideal solutions [2], SRs have a key role in environmental policy-making [4] and throughout the policy-making cycle (Fig. 1).

Fig. 1

(adapted from [32, 33])

The use of SR methods in the policy-making cycle, white boxes identify potential input of reviews

Despite the increasing awareness of the benefits of SRs in the researcher community, there have been calls to increase the use of evidence synthesis in policy-making [5, 6]. As a result, this commentary piece builds on the recent special edition on stakeholder engagement in environmental evidence synthesis [17] to introduce the challenges of working with policy-makers on SRs in the area of environmental management/conservation for those who may be new to this or those who wish to try to increase the use of SRs within policy-making. Using our experiences of producing SRs with Government Departments and Agencies within the UK and Europe we also, identify possible solutions and share our lessons learned.

The paper primarily focuses on demand-led SRs, i.e. those that are driven by the users of the review, but where possible and relevant we have also included suggestions for supply-led reviews that are driven by the researchers/research community. We have used the SR methodological process to identify challenges and solutions at each stage of the review process.

Pre-review challenges and solutions

The first challenge of using SRs in environmental policy is that the time and resources required for their completion can prohibit their use. In some instances, the costs required for a SR can be met by explaining the benefits a full SR provides, e.g. the increased rigor and the comprehensive nature of the review is likely to lead to more accurate findings, representing a sound investment [3]. This is especially important where decisions affect further resource allocation, numerous stakeholders, are high profile or controversial, and are likely to be subject to in-depth scrutiny.

However, timeliness can be a critical factor for policy-making and by not working to decision makers’ timelines, reviews may not be considered and used in the policy-making process [7]. A pragmatic approach may need to be taken where the rigour of best practice in SRs is not possible. Similar to the approach taken by Langer et al. [7], the UK Civil Service has adapted SR methodology to other methods which utilise systematic principles but are less resource and time intensive [8]. Whilst doing this may reduce a review’s robustness and reliability, it is preferable to un-structured reviews that do not apply systematic principles or not using evidence at all. Others have also identified alternative methods for evidence synthesis that could be used by decision makers and discussed their selection and use [9, 10].

In our experience, a risk-based approach can identify the most appropriate method by taking into account how the outputs of the review will be used. Policy decisions of high consequence should always be supported by the more robust SR methods as outputs will need to withstand a high-level of scrutiny. More rapid methods may be more appropriate for scoping work or where there is no time for full SR [8]. Communicating the trade-off between the degree of the application of systematic methods and timeliness of reviews to policy-makers ensures the risk associated with the selected review method is transparent and understood. Furthermore, close collaboration on the identification of the method is likely to increase legitimacy as the syntheses are recognised as having responded to the priorities and values of the users [7, 11].

Defining the question and scope

A key challenge for SRs, as often with other types of research, is that they may be seen as not addressing relevant questions for policy-makers. This means the review may not help to inform policy-making but could also damage the reputation of SRs as these may then not be considered as useful. Ensuring that questions of direct relevance to decision makers are addressed by SRs has been identified as significantly enhancing the benefits of SRs [12]. A method to overcome this challenge is to work with policy-makers to co-develop the review question and scope. By recognising the complex, multi-faceted and dynamic relationships between science-based knowledge and decision-making, co-production and the two-way exchange of information has been identified as key to improving the use of science within decision making [13,14,15]. Furthermore, active, iterative and inclusive communication is crucial to ensure the saliency, credibility and legitimacy of research with decision makers, which has been found to be key to the mobilisation of knowledge for action [16].

In a SR co-production can assist with understanding the context of the review and refining the scope and inclusion criteria, similar to engagement with other stakeholders [17]. Whilst this engagement may incur additional time and resources for reviewers, it will aid the uptake of the results of the review by ensuring buy-in from policy-makers who are then more likely to accept review findings and take these into consideration during the decision-making process.

Co-production can be seen as a continuum [18] so that where full co-production throughout the review is not an option, attempts to improve the relevancy of the SR question can still be made. At the lower end of the co-production continuum policy documents that outline statutory requirements, political commitments or evidence needs and strategies (e.g. [5]) and conferences (e.g. [6] provide an opportunity to identify potentially relevant SR topics and discuss evidence needs. More moderate levels of coproduction could involve engaging policy-makers to provide feedback on SR questions or draft protocols.

Policy-makers may make an evidence request that is not in a useable format for a SR. Policy-makers often ask questions regarding the best course of action, but reviews should not make value based judgements which could be biased. Instead, SRs can present evidence on effectiveness or impacts, so requests may need to be reworked into a format that can support policy-making [8]. Often initial requests for review topics from policy-makers will be broad and unsuitable for a SR. Working with policy-makers to identify the Population, Intervention/Exposure, Comparator and Outcome elements related to a question can help to create a question that is clear and focused [8, 19, 20]. This can ensure that the question and search terms are of direct relevance, for example defining a more focused geographic context which has been identified as potentially increasing the benefits of SRs [12].

Policy-makers often require information on cost-effectiveness in order to make informed decisions. This is often due to the limited resources for environmental and conservation management [9] and the need for accountability and value for money to taxpayers. In health care research systems have been developed to consider resource use and to rate the quality of economic evidence [21]. Whilst a number of challenges of applying these to conservation and environmental management exist, the adaptation of these has been explored and provides an opportunity to ensure SRs are relevant with policy-makers [9]. Therefore, reviewers should consider incorporating resource use into the scope for SRs and incorporate cost data into the searching and synthesis stages.

Another challenge in developing the question and scope of a policy relevant SR is differences in understanding and language used between policy-makers and reviewers. This can lead to miscommunications which can result in the review’s findings being unusable or delays in the completion of the review. For example, a review conducted by one of the authors of this paper on slurry storage originally considered the physical design of storage but in fact policy-makers required information on environmental implications of the practice [22]. Co-developing conceptual models, which present the focus of the review as a schematic or system diagram (e.g. James et al. [23]), can be particularly useful to ensure shared understanding of the scope and make explicit any technical terms and assist in developing a theory of change, describing how and why the change investigated by the review is expected to happen [24].

Developing protocols and undertaking the review

A key challenge may be that time and resources to publish a protocol through an external peer-reviewed process are not available. This can be overcome by working with policy-makers to ensure they understand that a protocol is a key requirement of a rigorous process and that the findings of a comprehensive and unbiased review will be more reliable. In our experience, whilst policy-makers recognise that peer-review provides independent assessment and acts as a quality stamp that helps wider acceptance, cost and time requirements are often prohibitive. Factoring peer-review into resource planning can help, but in our experience this can be hard to justify for policy-makers. Instead peer-review is often conducted by members of the steering group and expert advisors, who are often from external organisations. Whilst these quality assurance processes are different to traditional SRs, they may not be any less stringent [7]. For example, if a review influences a decision that could have a large impact on stakeholders this may be scrutinised internally by civil servants and ministers, by other members of parliament, select committees and potentially the media, external organisations and the wider public.

As with other SRs the amounts of evidence found by a policy relevant review can be problematic. This can occur when a large a volume of search results is found which cannot be processed and synthesised. This can be overcome by trialling search terms, in line with CEE guidelines, to identify the optimal search strings [19]. Working with policy-makers can reduce the scope of review in a systematic manner (e.g. relevant climate zone, date restrictions), to ensure that volumes of evidence are manageable and relevant. Where this is done, the rationale should be transparently recorded in the protocol and review. Another challenge is when too little evidence is found by a review. Similarly to working with other stakeholders, policy-makers can provide access to studies that are published informally (e.g. grey literature) such as government and consultancy reports, along with practitioner-held information [17]. Working with subject experts can also assist with this, as can broadening the forms of intervention and outcomes searched for to increase the data available [25]. However, there may still be little evidence found, which will be a frustration for policy-makers. Whilst there is no immediate fix to this, finding small amounts or no evidence can help to identify where there is little support for policy ideas, thus managing expectations and making risks of decisions known. It can also highlight knowledge gaps which, through close working with researchers and funders, can lead to commissioning of policy relevant primary research.

Critically appraising evidence may be a challenge as policy relevant SRs often make use of diverse types of evidence. In environmental policy, policy-makers will be concerned not only with natural and physical processes, but also cost-benefit information, how people interact with the environment, their impact on it and their behaviour. Qualitative research and economic appraisals may be relevant alongside field or laboratory studies. Greater use of existing knowledge that includes a broader range of evidence types has been identifies as an element in enhancing the benefits of SRs to decision making [12]. Therefore, critical appraisal that allows for comparison and assessment of these varied study designs is necessary [8]. Appraisal should consider individual methodological application and the mitigation of bias relevant to the study design type in all evidence sources identified [8, 26].

Synthesising and communicating SR results

Synthesising across the diverse range of evidence often found in policy-relevant environmental SRs is a challenge. Lack of similar studies often makes meta-analysis of findings unwise in policy-relevant environmental SRs. As a result narrative synthesis is often relied upon which can be useful for decision-makers. Bias in the summary should be reduced by discussing the volume and characteristics, such as study types and aspects of study quality of all the evidence found by the review and using tables and graphical descriptions to support syntheses [27].

Additionally, the lack of randomisation, allocation concealment and blinding in the studies often found by policy-relevant environmental/conservation SRs has meant that tools such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system used to assess the overall quality of a body of evidence in healthcare [28] are not always appropriate. However, we have found that the principles can be applied to some degree and it is particularly important to include statements that describe evidence bodies as consistent/contested (where one or more studies findings conflict with others) or mixed (where diverse studies, applied in a range of contexts, have produced contrasting results) [29]. Assigning confidence to particular statements is particularly useful for decision-makers. This should involve results of critical appraisals [8] so that simple vote-counting is avoided due to the large variation in the studies of used in policy-relevant SRs.

Presenting the results of a SR in a relevant manner so that the findings of the review are to be considered in the policy-making process can be a challenge. In our experience, the best way to maximise impact with policy-makers is to provide a range of communication products. In addition to the full report, a non-technical Executive Summary (maximum two pages) helps ensure findings are readily understood by end-users, whilst maintaining transparency of methods used. Policy summaries, posters and infographics may also be useful for communicating with more diverse audiences.

Full reports should communicate the evidence synthesis results, but also summarise the findings for each stage of the process, together with the methodology used and any deviation from the original protocol.

Reports should describe the evidence base and its findings but they should not make policy recommendations. Whilst reviews provide information and support to those making decisions, they should not advocate a particular decision, as this may risk undermining scientific integrity [30].

Again we have found that co-production assists in ensuring that reviews’ findings are considered in the policy-making process. This is because engagement throughout the process assists with decision-makers developing a sense of ownership in the research and a strong understanding of the research content, which they can then communicate more broadly within their organisation [13].

SRs may produce unexpected or controversial findings. As a result of this it is essential that policy-makers and other stakeholders are involved throughout the review process and that evidence from all sources is sought. In our experience, asking stakeholders to suggest evidence they feel is relevant can help ensure no key information is missed, whilst harbouring a sense of inclusivity in the process. However, this evidence must be subject to the same screening and scrutiny processes as all other evidence.

A final challenge for SRs is that findings appear not to be used within the policy-making process causing frustration for reviewers. It should be recognised that policy-making can be a long process and does not always involve legislation and regulation. Therefore, the findings of the review may have fed into earlier, more formative steps in the policy-making cycle. Furthermore, where policy-makers are engaged in SRs they may develop related skills [7] and awareness, increasing demand for systematic processes in future which could lead to changes in how evidence is requested and viewed in policy-making.

Conclusion

Whilst a number of challenges of engaging environmental policy-makers throughout the SR process have been identified, this paper has highlighted potential solutions and shared lessons learned to address these (Table 1).

Table 1 Challenges identified with engaging policy-makers with the SR process and potential solutions

Numerous challenges of using SRs in environmental policy can be overcome by a co-development process which has been identified as key to improving the use of science within decision making [8,9,15, 31]. Within SRs a co-develop approach can help to ensure that SR questions are relevant, that the context of the review is taken into consideration and can help with the communication of the results, ensuring that their importance in decision-making is recognised by policy-makers. As with other areas of knowledge transfer this active, iterative, and inclusive communication is crucial to ensure saliency, credibility and legitimacy so that knowledge is mobilised for action (Cash et al. 2003). Without this, SRs may not be seen as valuable by decision-makers, potentially damaging their reputation and preventing further use. Furthermore, co-development can help develop skills in both reviewers and policy-makers and can help to raise appreciation of systematic methods, leading to changes in practice and culture within decision-making organisations. There may be concerns regarding close working arrangements and impartiality of reviews, so it is essential that they key tenets of SRs (objectivity, rigour, replicability and transparency) are upheld at all times.

The largest challenge for SRs is the time and cost taken to undertake them. Close working with policy-makers allows the benefits of the SR process to be understood, and using a risk-based approach to selecting the review method can help to communicate the trade-offs associated with method selection. Where a full SR is not possible then the authors believe that a pragmatic approach to applying systematic methods should be taken, to ensure that reviews are responsive to policy-makers’ needs and to facilitate the use of evidence in policy-making. Further discussion and the development of a community of practice of reviewers engaged with policy-makers will help to further explore methods to overcome the challenges identified and the creation of best-practices.