Skip to content

Advertisement

  • Commentary
  • Open Access

Engaging environmental policy-makers with systematic reviews: challenges, solutions and lessons learned

Environmental Evidence20198:2

https://doi.org/10.1186/s13750-018-0144-0

  • Received: 12 January 2018
  • Accepted: 20 December 2018
  • Published:

Abstract

The creation and accumulation of robust bodies of knowledge, along with their dissemination, utilisation and integration in decision support are key to improving the use of evidence in decision-making. Systematic reviews (SRs), through their emphasis on transparency, replicability and rigour, offer numerous benefits throughout the policy-making cycle and for improving the use of evidence in environmental policy-making. As a result there have been numerous calls to increase the use of SRs in environmental policy-making. This commentary paper introduces the challenges of engaging policy-makers with SRs and, using experiences of producing SRs with Government Departments and Agencies within the UK and Europe, identifies possible solutions and shares our lessons learned. It highlights that co-production can help to overcome a number of challenges by ensuring that review questions are policy-relevant, that the context of the review is taken into consideration and that review’s findings are communicated so that they are recognised and used in policy decision-making processes. Additionally, a pragmatic approach to the review’s methodology may be required to respond to policy-making requirements. Here, a risk-based approach can communicate the trade-offs between the rigour and timeliness of the review. Ensuring that systematic approaches are upheld at all times can help address impartiality concerns and can develop skills in both reviewers and policy-makers to increase awareness of systematic methods, leading to changes in practice and culture within decision-making organisations and the promotion of evidence informed policy development and decisions.

Introduction

The creation and accumulation of robust bodies of knowledge, along with their dissemination, utilisation and integration in decision support are key to improving the use of evidence in decision-making [1]. Systematic reviews (SRs), through emphasis on objectivity, rigour, replicability and transparency offer numerous benefits in the policy-making process and will be particularly important where outcomes may influence high-impact decisions that affects those who have either a direct or indirect a stake, known hereafter as stakeholders [2], or in areas that may be contested and controversial [3]. As environmental management is characterised by complexity, with multiple stakeholders and uncertain ideal solutions [2], SRs have a key role in environmental policy-making [4] and throughout the policy-making cycle (Fig. 1).
Fig. 1
Fig. 1

The use of SR methods in the policy-making cycle, white boxes identify potential input of reviews

(adapted from [32, 33])

Despite the increasing awareness of the benefits of SRs in the researcher community, there have been calls to increase the use of evidence synthesis in policy-making [5, 6]. As a result, this commentary piece builds on the recent special edition on stakeholder engagement in environmental evidence synthesis [17] to introduce the challenges of working with policy-makers on SRs in the area of environmental management/conservation for those who may be new to this or those who wish to try to increase the use of SRs within policy-making. Using our experiences of producing SRs with Government Departments and Agencies within the UK and Europe we also, identify possible solutions and share our lessons learned.

The paper primarily focuses on demand-led SRs, i.e. those that are driven by the users of the review, but where possible and relevant we have also included suggestions for supply-led reviews that are driven by the researchers/research community. We have used the SR methodological process to identify challenges and solutions at each stage of the review process.

Pre-review challenges and solutions

The first challenge of using SRs in environmental policy is that the time and resources required for their completion can prohibit their use. In some instances, the costs required for a SR can be met by explaining the benefits a full SR provides, e.g. the increased rigor and the comprehensive nature of the review is likely to lead to more accurate findings, representing a sound investment [3]. This is especially important where decisions affect further resource allocation, numerous stakeholders, are high profile or controversial, and are likely to be subject to in-depth scrutiny.

However, timeliness can be a critical factor for policy-making and by not working to decision makers’ timelines, reviews may not be considered and used in the policy-making process [7]. A pragmatic approach may need to be taken where the rigour of best practice in SRs is not possible. Similar to the approach taken by Langer et al. [7], the UK Civil Service has adapted SR methodology to other methods which utilise systematic principles but are less resource and time intensive [8]. Whilst doing this may reduce a review’s robustness and reliability, it is preferable to un-structured reviews that do not apply systematic principles or not using evidence at all. Others have also identified alternative methods for evidence synthesis that could be used by decision makers and discussed their selection and use [9, 10].

In our experience, a risk-based approach can identify the most appropriate method by taking into account how the outputs of the review will be used. Policy decisions of high consequence should always be supported by the more robust SR methods as outputs will need to withstand a high-level of scrutiny. More rapid methods may be more appropriate for scoping work or where there is no time for full SR [8]. Communicating the trade-off between the degree of the application of systematic methods and timeliness of reviews to policy-makers ensures the risk associated with the selected review method is transparent and understood. Furthermore, close collaboration on the identification of the method is likely to increase legitimacy as the syntheses are recognised as having responded to the priorities and values of the users [7, 11].

Defining the question and scope

A key challenge for SRs, as often with other types of research, is that they may be seen as not addressing relevant questions for policy-makers. This means the review may not help to inform policy-making but could also damage the reputation of SRs as these may then not be considered as useful. Ensuring that questions of direct relevance to decision makers are addressed by SRs has been identified as significantly enhancing the benefits of SRs [12]. A method to overcome this challenge is to work with policy-makers to co-develop the review question and scope. By recognising the complex, multi-faceted and dynamic relationships between science-based knowledge and decision-making, co-production and the two-way exchange of information has been identified as key to improving the use of science within decision making [1315]. Furthermore, active, iterative and inclusive communication is crucial to ensure the saliency, credibility and legitimacy of research with decision makers, which has been found to be key to the mobilisation of knowledge for action [16].

In a SR co-production can assist with understanding the context of the review and refining the scope and inclusion criteria, similar to engagement with other stakeholders [17]. Whilst this engagement may incur additional time and resources for reviewers, it will aid the uptake of the results of the review by ensuring buy-in from policy-makers who are then more likely to accept review findings and take these into consideration during the decision-making process.

Co-production can be seen as a continuum [18] so that where full co-production throughout the review is not an option, attempts to improve the relevancy of the SR question can still be made. At the lower end of the co-production continuum policy documents that outline statutory requirements, political commitments or evidence needs and strategies (e.g. [5]) and conferences (e.g. [6] provide an opportunity to identify potentially relevant SR topics and discuss evidence needs. More moderate levels of coproduction could involve engaging policy-makers to provide feedback on SR questions or draft protocols.

Policy-makers may make an evidence request that is not in a useable format for a SR. Policy-makers often ask questions regarding the best course of action, but reviews should not make value based judgements which could be biased. Instead, SRs can present evidence on effectiveness or impacts, so requests may need to be reworked into a format that can support policy-making [8]. Often initial requests for review topics from policy-makers will be broad and unsuitable for a SR. Working with policy-makers to identify the Population, Intervention/Exposure, Comparator and Outcome elements related to a question can help to create a question that is clear and focused [8, 19, 20]. This can ensure that the question and search terms are of direct relevance, for example defining a more focused geographic context which has been identified as potentially increasing the benefits of SRs [12].

Policy-makers often require information on cost-effectiveness in order to make informed decisions. This is often due to the limited resources for environmental and conservation management [9] and the need for accountability and value for money to taxpayers. In health care research systems have been developed to consider resource use and to rate the quality of economic evidence [21]. Whilst a number of challenges of applying these to conservation and environmental management exist, the adaptation of these has been explored and provides an opportunity to ensure SRs are relevant with policy-makers [9]. Therefore, reviewers should consider incorporating resource use into the scope for SRs and incorporate cost data into the searching and synthesis stages.

Another challenge in developing the question and scope of a policy relevant SR is differences in understanding and language used between policy-makers and reviewers. This can lead to miscommunications which can result in the review’s findings being unusable or delays in the completion of the review. For example, a review conducted by one of the authors of this paper on slurry storage originally considered the physical design of storage but in fact policy-makers required information on environmental implications of the practice [22]. Co-developing conceptual models, which present the focus of the review as a schematic or system diagram (e.g. James et al. [23]), can be particularly useful to ensure shared understanding of the scope and make explicit any technical terms and assist in developing a theory of change, describing how and why the change investigated by the review is expected to happen [24].

Developing protocols and undertaking the review

A key challenge may be that time and resources to publish a protocol through an external peer-reviewed process are not available. This can be overcome by working with policy-makers to ensure they understand that a protocol is a key requirement of a rigorous process and that the findings of a comprehensive and unbiased review will be more reliable. In our experience, whilst policy-makers recognise that peer-review provides independent assessment and acts as a quality stamp that helps wider acceptance, cost and time requirements are often prohibitive. Factoring peer-review into resource planning can help, but in our experience this can be hard to justify for policy-makers. Instead peer-review is often conducted by members of the steering group and expert advisors, who are often from external organisations. Whilst these quality assurance processes are different to traditional SRs, they may not be any less stringent [7]. For example, if a review influences a decision that could have a large impact on stakeholders this may be scrutinised internally by civil servants and ministers, by other members of parliament, select committees and potentially the media, external organisations and the wider public.

As with other SRs the amounts of evidence found by a policy relevant review can be problematic. This can occur when a large a volume of search results is found which cannot be processed and synthesised. This can be overcome by trialling search terms, in line with CEE guidelines, to identify the optimal search strings [19]. Working with policy-makers can reduce the scope of review in a systematic manner (e.g. relevant climate zone, date restrictions), to ensure that volumes of evidence are manageable and relevant. Where this is done, the rationale should be transparently recorded in the protocol and review. Another challenge is when too little evidence is found by a review. Similarly to working with other stakeholders, policy-makers can provide access to studies that are published informally (e.g. grey literature) such as government and consultancy reports, along with practitioner-held information [17]. Working with subject experts can also assist with this, as can broadening the forms of intervention and outcomes searched for to increase the data available [25]. However, there may still be little evidence found, which will be a frustration for policy-makers. Whilst there is no immediate fix to this, finding small amounts or no evidence can help to identify where there is little support for policy ideas, thus managing expectations and making risks of decisions known. It can also highlight knowledge gaps which, through close working with researchers and funders, can lead to commissioning of policy relevant primary research.

Critically appraising evidence may be a challenge as policy relevant SRs often make use of diverse types of evidence. In environmental policy, policy-makers will be concerned not only with natural and physical processes, but also cost-benefit information, how people interact with the environment, their impact on it and their behaviour. Qualitative research and economic appraisals may be relevant alongside field or laboratory studies. Greater use of existing knowledge that includes a broader range of evidence types has been identifies as an element in enhancing the benefits of SRs to decision making [12]. Therefore, critical appraisal that allows for comparison and assessment of these varied study designs is necessary [8]. Appraisal should consider individual methodological application and the mitigation of bias relevant to the study design type in all evidence sources identified [8, 26].

Synthesising and communicating SR results

Synthesising across the diverse range of evidence often found in policy-relevant environmental SRs is a challenge. Lack of similar studies often makes meta-analysis of findings unwise in policy-relevant environmental SRs. As a result narrative synthesis is often relied upon which can be useful for decision-makers. Bias in the summary should be reduced by discussing the volume and characteristics, such as study types and aspects of study quality of all the evidence found by the review and using tables and graphical descriptions to support syntheses [27].

Additionally, the lack of randomisation, allocation concealment and blinding in the studies often found by policy-relevant environmental/conservation SRs has meant that tools such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system used to assess the overall quality of a body of evidence in healthcare [28] are not always appropriate. However, we have found that the principles can be applied to some degree and it is particularly important to include statements that describe evidence bodies as consistent/contested (where one or more studies findings conflict with others) or mixed (where diverse studies, applied in a range of contexts, have produced contrasting results) [29]. Assigning confidence to particular statements is particularly useful for decision-makers. This should involve results of critical appraisals [8] so that simple vote-counting is avoided due to the large variation in the studies of used in policy-relevant SRs.

Presenting the results of a SR in a relevant manner so that the findings of the review are to be considered in the policy-making process can be a challenge. In our experience, the best way to maximise impact with policy-makers is to provide a range of communication products. In addition to the full report, a non-technical Executive Summary (maximum two pages) helps ensure findings are readily understood by end-users, whilst maintaining transparency of methods used. Policy summaries, posters and infographics may also be useful for communicating with more diverse audiences.

Full reports should communicate the evidence synthesis results, but also summarise the findings for each stage of the process, together with the methodology used and any deviation from the original protocol.

Reports should describe the evidence base and its findings but they should not make policy recommendations. Whilst reviews provide information and support to those making decisions, they should not advocate a particular decision, as this may risk undermining scientific integrity [30].

Again we have found that co-production assists in ensuring that reviews’ findings are considered in the policy-making process. This is because engagement throughout the process assists with decision-makers developing a sense of ownership in the research and a strong understanding of the research content, which they can then communicate more broadly within their organisation [13].

SRs may produce unexpected or controversial findings. As a result of this it is essential that policy-makers and other stakeholders are involved throughout the review process and that evidence from all sources is sought. In our experience, asking stakeholders to suggest evidence they feel is relevant can help ensure no key information is missed, whilst harbouring a sense of inclusivity in the process. However, this evidence must be subject to the same screening and scrutiny processes as all other evidence.

A final challenge for SRs is that findings appear not to be used within the policy-making process causing frustration for reviewers. It should be recognised that policy-making can be a long process and does not always involve legislation and regulation. Therefore, the findings of the review may have fed into earlier, more formative steps in the policy-making cycle. Furthermore, where policy-makers are engaged in SRs they may develop related skills [7] and awareness, increasing demand for systematic processes in future which could lead to changes in how evidence is requested and viewed in policy-making.

Conclusion

Whilst a number of challenges of engaging environmental policy-makers throughout the SR process have been identified, this paper has highlighted potential solutions and shared lessons learned to address these (Table 1).
Table 1

Challenges identified with engaging policy-makers with the SR process and potential solutions

Stage of the review

Challenge identified

Potential solutions

Pre-review

Prohibitive time and cost of producing a SR

Explain benefits of rigorous and comprehensive review to policy-maker and the trade-off less rigorous methods involve. This is especially important where the evidence base is diverse or controversial and where multiple stakeholders are affected or the decision is likely to be of high consequence and subject to high profile and in-depth scrutiny

Pragmatic approaches to applying systematic review standards may be required to reduce the scope and scale of a search to enable systematic approaches to respond to policy-makers’ needs e.g. [8]. Here a risk-based approach should be applied

Close collaboration with decision makers on the identification of the most appropriate method is likely to increase legitimacy as the syntheses are recognised as having responded to the priorities and values of the users

Defining the question and scope of the review

Question addressed by SR not relevant to policy

Co-development of question and scope of the review between reviewers and policy-makers

Active, iterative, and inclusive communication between policy-makers and review teams to ensure the saliency, credibility and legitimacy of the review

Understand policy/statutory context and anticipate evidence needs. This can be done thorough consultation of policy documents that outline statutory requirements, political commitments or evidence needs and strategies and policy conferences

Question posed by policy-makers not in a useable format, e.g. too broad or too vague, or leading in nature

Allow time to revise the question into a useable format. Ensure that policy-makers understand that the review can only describe evidence, it cannot make a value judgement

Work with policy-makers to identify the Population, Intervention/Exposure, Comparator and Outcome elements related to a question can help to create a question that is clear and focused

Policy-makers often require information on cost-effectiveness in order to make informed decisions

Develop approaches to incorporate cost-effectiveness information and the critical appraisal of this into the scope of the review. Approach could be adapted from health care research

Differences in understanding of the context of the review and miscommunication of terms, can be especially apparent between policy-makers and reviewers

Develop conceptual models to ensure shared understanding and boundaries of the review are well defined and to make explicit any technical terms

Developing the protocol and undertaking the review

Time and resources required for the publication of an external peer reviewed protocol are not made available

Hold conversations to ensure that policy-makers understand the benefit of peer review e.g. independent assessment and stamp of approval to help wider acceptance

Build adequate cost and time into project

Where these are not possible then peer review by the steering group and relevant external experts will have to be used

Challenging amounts of evidence, i.e. too much or too little relevant evidence

Allow adequate time to refine search strings to ensure relevancy

If necessary work with policy-makers and subject experts to reduce scope of review in a systematic manner e.g. relevant climate zone. Ensure that where this is done it is transparently recorded in the protocol and review report

Work with policy-makers to identify potentially relevant grey literature e.g. government and consultancy reports and practitioner information

Too little relevant evidence found can be particularly challenge. However this should be used to identify knowledge gaps, which can be valuable to research funders. This can be used to engage policy-makers with further primary research

Critical appraise of diverse evidence types is required

Design critical appraisal methods to enable diverse types of evidence, such as qualitative studies, structured interviews etc. to be incorporated. This should consider individual methodological application and the mitigation of bias relevant to the study design

 

Synthesising across the diverse range of evidence types

Narrative summaries are particularly useful in policy but here synthesis should detail the volume and characteristics of the evidence found. Communicating the degree of consistency or uncertainty in the evidence will also be of use to policy

Describe evidence bodies as consistent/contested (where one or more studies findings conflict with others) or mixed (where diverse studies, applied in a range of contexts, have produced contrasting results). Used the results of critical appraisal to describe the degrees of certainty in the evidence findings

Communication of results

Communication of results in a policy relevant manner

A range of communication products may be required, including a 2-page executive summary along with a full report detailing methodological details for transparency. Infographics may also be of use

Co-production assists in ensuring that reviews’ findings are considered in the policy-making process due to decision-makers developing a sense of ownership in the research and a strong understanding of the research content

Controversial or politicised findings

This will sometimes be unavoidable but engagement of policy-makers along with other stakeholders throughout the process can mitigate this

Requesting evidence and search locations from a wide stakeholder base can increase trust and inclusivity in the review. However, this evidence must be subject to the same screening and scrutiny processes as all other evidence

Results not used in the policy decision-makers process causing frustration for reviewers

Recognition that policy-making does not always involve legislation or regulation and work may feed into a longer or formative and scoping process

Engagement presents an opportunity to develop skills by both reviewers and policy-makers and can increase awareness and demand of systematic processes in policy

Numerous challenges of using SRs in environmental policy can be overcome by a co-development process which has been identified as key to improving the use of science within decision making [815, 31]. Within SRs a co-develop approach can help to ensure that SR questions are relevant, that the context of the review is taken into consideration and can help with the communication of the results, ensuring that their importance in decision-making is recognised by policy-makers. As with other areas of knowledge transfer this active, iterative, and inclusive communication is crucial to ensure saliency, credibility and legitimacy so that knowledge is mobilised for action (Cash et al. 2003). Without this, SRs may not be seen as valuable by decision-makers, potentially damaging their reputation and preventing further use. Furthermore, co-development can help develop skills in both reviewers and policy-makers and can help to raise appreciation of systematic methods, leading to changes in practice and culture within decision-making organisations. There may be concerns regarding close working arrangements and impartiality of reviews, so it is essential that they key tenets of SRs (objectivity, rigour, replicability and transparency) are upheld at all times.

The largest challenge for SRs is the time and cost taken to undertake them. Close working with policy-makers allows the benefits of the SR process to be understood, and using a risk-based approach to selecting the review method can help to communicate the trade-offs associated with method selection. Where a full SR is not possible then the authors believe that a pragmatic approach to applying systematic methods should be taken, to ensure that reviews are responsive to policy-makers’ needs and to facilitate the use of evidence in policy-making. Further discussion and the development of a community of practice of reviewers engaged with policy-makers will help to further explore methods to overcome the challenges identified and the creation of best-practices.

Declarations

Authors’ contributions

This paper is based on a draft written by AMC. DC and NR assisted in editing and revising the draft. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Funding

AMC received a Policy Placement Fellowship funded by the Natural Environment Research Council (Grant Number NE/L011506/1), the UK Department for Environment Food and Rural Affairs and the Environment Agency, which enabled the development of the ideas presented in this paper.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Centre for Environmental Policy, Imperial College London, 15 Prince’s Gardens, South Kensington, London, SW7 1NA, UK
(2)
Centre for Evidence Based Agriculture, Department of Crop and Environment Sciences, Harper Adams University, Newport, Shropshire, TF10 8NB, UK

References

  1. Nutley S, Walter I, Davies HTO. From knowing to doing: a framework for understanding the evidence-into-practice agenda. Evaluation. 2003;9(2):125–48. https://doi.org/10.1177/1356389003009002002.View ArticleGoogle Scholar
  2. Reed MS. Stakeholder participation for environmental management: A literature review. Biol Conserv. 2008;141(10):2417–31. https://doi.org/10.1016/j.biocon.2008.07.014.View ArticleGoogle Scholar
  3. Haddaway NR, Woodcock P, Macura B, Collins A. Making literature reviews more reliable through application of lessons from systematic reviews. Conserv Biol. 2015;29(6):1596–5. https://doi.org/10.1111/cobi.12541.View ArticleGoogle Scholar
  4. Bilotta GS, Milner AM, Boyd I. On the use of systematic reviews to inform environmental policies. Environ Sci Policy 2014;42:67–77.View ArticleGoogle Scholar
  5. Defra. Making the most of our evidence: a strategy for Defra and its network. 2014. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/318610/evidence-strategy-defra.pdf. Accessed 17 Aug 2017.
  6. The Royal Society. Science for Defra: excellence in the application of evidence. 2017. https://royalsociety.org/science-events-and-lectures/2017/03/defra-conference/. Accessed 3 Sept 2018.
  7. Langer L, Erasmus Y, Tannous N, Stewart R. How stakeholder engagement has led us to reconsider definitions of rigour in systematic reviews. Environ Evid. 2017;6(1):20. https://doi.org/10.1186/s13750-017-0098-7.View ArticleGoogle Scholar
  8. Collins A, Coughlin D, Miller J, Kirk S. The production of quick scoping reviews and rapid evidence assessments: a how to guide. 2015. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/560521/Production_of_quick_scoping_reviews_and_rapid_evidence_assessments.pdf. Accessed 3 Sept 2018.
  9. Cook CN, et al. Simplifying the selection of evidence synthesis methods to inform environmental decisions: a guide for decision makers and scientists. Biol Conserv. 2017;213:135–45.View ArticleGoogle Scholar
  10. Dicks LV, Haddaway N, Hernández-Morcillo M, Mattsson B, Randall N, Failler P, Ferretti J, Livoreil B, Saarikoski H, Santamaria L, Rodela R, Velizarova E. Knowledge synthesis for environmental decisions: an evaluation of existing methods, and guidance for their selection, use and development. 2017. http://www.eklipse-mechanism.eu/apps/Eklipse_data/website/EKLIPSE_D3-1-Report_FINAL_WithCovers_V6.pdf. Accessed 3 Sept 2018.
  11. Parkhurst J. The Politics of Evidence : From evidence-based policy to the good governance of evidence. London: Taylor & Francis; 2017. Retrieved from http://www.oapen.org/search?identifier=640550.Google Scholar
  12. Cook CN, Possingham HP, Fuller RA. Contribution of systematic reviews to management decisions. Conserv Biol. 2013;27(5):902–15.View ArticleGoogle Scholar
  13. Cvitanovic C, et al. Improving knowledge exchange among scientists and decision-makers to facilitate the adaptive governance of marine resources: a review of knowledge and research needs. Ocean Coast Manag. 2015;112:25–35.View ArticleGoogle Scholar
  14. van Kerkhoff L, Lebel L. Linking knowledge and action for sustainable development. Ann Rev Environ Resour. 2006;31(1):445–77. https://doi.org/10.1146/annurev.energy.31.102405.170850.View ArticleGoogle Scholar
  15. Roux DJ, et al. Bridging the science-management divide: moving from unidirectional knowledge transfer to knowledge interfacing and sharing. Ecol Soc. 2006;11(1). https://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid=127797. Accessed 30 Aug 2018.
  16. Cash DW, Clark WC, Alcock F, Dickson NM, Eckley N, Guston DH, Jäger J, Mitchell RB. Knowledge systems for sustainable development. Proc Nat Acad Sci U.S.A. 2003;100(14):8086–91. https://doi.org/10.1073/pnas.1231332100.View ArticleGoogle Scholar
  17. Haddaway NR, Kohl C, Rebelo da Silva N, Schiemann J, Spök A, Stewart R., Sweet JB, Wilhelm R. A framework for stakeholder engagement during systematic reviews and maps in environmental management. Environ Evid. 2017;6(1):11. https://doi.org/10.1186/s13750-017-0089-8.View ArticleGoogle Scholar
  18. Buick F, et al. Effective practitioner–scholar relationships: lessons from a coproduction partnership. Public Adm Rev. 2016;76(1):35–47.View ArticleGoogle Scholar
  19. Collaboration for Environmental Evidence (CEE). Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. 2013. http://www.environmentalevidence.org/Documents/Guidelines/Guidelines4.2.pdf. Accessed 3 Sept 2018
  20. Guyatt GH, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64(4):395–400.View ArticleGoogle Scholar
  21. Brunetti M, et al. GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence. J Clin Epidemiol. 2013;66(2):140–50.View ArticleGoogle Scholar
  22. Randall NP, Donnison LM, Lewis PJ, James KL. How effective are on-farm mitigation measures for delivering an improved water environment? A systematic map. Environ Evid. 2015;4:18. https://doi.org/10.1186/s13750-015-0044-5.View ArticleGoogle Scholar
  23. James K, Randall NP, Millington A. Quick Scoping Review on the impact of amenity pesticides on the water environment. 2014. Retrieved from http://randd.defra.gov.uk/Default.aspxMenu=Menu&Module=More%26Location=None%26ProjectID=19195%26FromSearch=Y%26Publisher=1%26SearchText=WT154%26SortString=ProjectCode%26SortOrder=Asc%26Paging=10#Description.
  24. Vogel I. Review of the use of ‘Theory of Change’ in international development. Department for International Development (DfID); 2012.Google Scholar
  25. Doerr ED, et al. Maximizing the value of systematic reviews in ecology when data or resources are limited. Austral Ecol. 2015;40(1):1–11. https://doi.org/10.1111/aec.12179.View ArticleGoogle Scholar
  26. Centre for Reviews and Dissemination (CRD). Systematic Reviews: CRD’s guidance for undertaking reviews in health care. 2009. Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf.
  27. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme version, 1. 2006.Google Scholar
  28. Guyatt GH, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.View ArticleGoogle Scholar
  29. DfID (Department for International Development). How to guide: assessing the strength of evidence. 2014. https://www.gov.uk/government/publications/how-to-note-assessing-the-strength-of-evidence. Accessed 3 Sept 2018.
  30. Boyd I. Making science count in government. eLife. 2013;2:e01061. http://elife.elifesciences.org/content/2/e01061. Accessed 3 Sept 2018.
  31. Karl HA, Susskind LE, Wallace KH. Environment: science and policy for sustainable development a dialogue, not a diatribe: effective integration of science and policy through joint fact finding. 2010.Google Scholar
  32. Office for Science. Guidelines on scientific analysis in policy making government response to the public consultation. 2010. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/207424/10-992-guidelines-scientific-analysis-policy-making-final-govt-response.pdf. Accessed 3 Sept 2018.
  33. Defra. Defra’s evidence investment strategy: 2010–2013 and beyond—2011 update. 2011. https://www.gov.uk/government/publications/defra-s-evidence-investment-strategy-2010-to-2013-and-beyond-2011-update. Accessed 3 Sept 2018.

Copyright

© The Author(s) 2019

Advertisement