Table 1 Challenges identified with engaging policy-makers with the SR process and potential solutions

Pre-review

Prohibitive time and cost of producing a SR

Explain benefits of rigorous and comprehensive review to policy-maker and the trade-off less rigorous methods involve. This is especially important where the evidence base is diverse or controversial and where multiple stakeholders are affected or the decision is likely to be of high consequence and subject to high profile and in-depth scrutiny

Pragmatic approaches to applying systematic review standards may be required to reduce the scope and scale of a search to enable systematic approaches to respond to policy-makers’ needs e.g. [8]. Here a risk-based approach should be applied

Close collaboration with decision makers on the identification of the most appropriate method is likely to increase legitimacy as the syntheses are recognised as having responded to the priorities and values of the users

Defining the question and scope of the review

Question addressed by SR not relevant to policy

Co-development of question and scope of the review between reviewers and policy-makers

Active, iterative, and inclusive communication between policy-makers and review teams to ensure the saliency, credibility and legitimacy of the review

Understand policy/statutory context and anticipate evidence needs. This can be done thorough consultation of policy documents that outline statutory requirements, political commitments or evidence needs and strategies and policy conferences

Question posed by policy-makers not in a useable format, e.g. too broad or too vague, or leading in nature

Allow time to revise the question into a useable format. Ensure that policy-makers understand that the review can only describe evidence, it cannot make a value judgement

Work with policy-makers to identify the Population, Intervention/Exposure, Comparator and Outcome elements related to a question can help to create a question that is clear and focused

Policy-makers often require information on cost-effectiveness in order to make informed decisions

Develop approaches to incorporate cost-effectiveness information and the critical appraisal of this into the scope of the review. Approach could be adapted from health care research

Differences in understanding of the context of the review and miscommunication of terms, can be especially apparent between policy-makers and reviewers

Develop conceptual models to ensure shared understanding and boundaries of the review are well defined and to make explicit any technical terms

Developing the protocol and undertaking the review

Time and resources required for the publication of an external peer reviewed protocol are not made available

Hold conversations to ensure that policy-makers understand the benefit of peer review e.g. independent assessment and stamp of approval to help wider acceptance

Build adequate cost and time into project

Where these are not possible then peer review by the steering group and relevant external experts will have to be used

Challenging amounts of evidence, i.e. too much or too little relevant evidence

Allow adequate time to refine search strings to ensure relevancy

If necessary work with policy-makers and subject experts to reduce scope of review in a systematic manner e.g. relevant climate zone. Ensure that where this is done it is transparently recorded in the protocol and review report

Work with policy-makers to identify potentially relevant grey literature e.g. government and consultancy reports and practitioner information

Too little relevant evidence found can be particularly challenge. However this should be used to identify knowledge gaps, which can be valuable to research funders. This can be used to engage policy-makers with further primary research

Critical appraise of diverse evidence types is required

Design critical appraisal methods to enable diverse types of evidence, such as qualitative studies, structured interviews etc. to be incorporated. This should consider individual methodological application and the mitigation of bias relevant to the study design

Synthesising across the diverse range of evidence types

Narrative summaries are particularly useful in policy but here synthesis should detail the volume and characteristics of the evidence found. Communicating the degree of consistency or uncertainty in the evidence will also be of use to policy

Describe evidence bodies as consistent/contested (where one or more studies findings conflict with others) or mixed (where diverse studies, applied in a range of contexts, have produced contrasting results). Used the results of critical appraisal to describe the degrees of certainty in the evidence findings

Communication of results

Communication of results in a policy relevant manner

A range of communication products may be required, including a 2-page executive summary along with a full report detailing methodological details for transparency. Infographics may also be of use

Co-production assists in ensuring that reviews’ findings are considered in the policy-making process due to decision-makers developing a sense of ownership in the research and a strong understanding of the research content

Controversial or politicised findings

This will sometimes be unavoidable but engagement of policy-makers along with other stakeholders throughout the process can mitigate this

Requesting evidence and search locations from a wide stakeholder base can increase trust and inclusivity in the review. However, this evidence must be subject to the same screening and scrutiny processes as all other evidence

Results not used in the policy decision-makers process causing frustration for reviewers

Recognition that policy-making does not always involve legislation or regulation and work may feed into a longer or formative and scoping process

Engagement presents an opportunity to develop skills by both reviewers and policy-makers and can increase awareness and demand of systematic processes in policy