What has led us to develop this approach?

We understand the need for rigour. We have conducted reviews for 3ie, CEE, Cochrane, Campbell, and the EPPI-Centre and so bring a wealth of methodological expertise to the challenge of balancing stakeholder engagement with the need for rigour. Our work has at times been supply-led and at other times demand-led, and this has influenced the range of people involved and types of engagement we have undertaken as well as how we have viewed the concept of rigour. We have worked with a wide range of stakeholders using approaches all along the spectrum of involvement [9]; from one-off requests for advice from stakeholders to formal advisory groups, working groups, and even full co-production. We have also supported decision-makers in producing their own evidence syntheses. In employing this spectrum of engagement approaches, we have produced a wide range of synthesis products: from full reviews through to responsive evidence assessments, reviews of reviews, and evidence maps.

The range of stakeholder engagement we have undertaken, with different drivers and different products, has led us to reflect that the definition of rigour commonly used by research producers does not always fit within particular stakeholder contexts, and therefore we have been reconsidering the question of what is a ‘gold standard’ systematic review.

An overview of the approach we now use

As a team committed to producing evidence syntheses which are demand-led, useful, and used we have to take seriously these issues about how and why to include stakeholders and how to address the tensions with regard to rigour that arise as a result. As methodologists we have a good understanding of the ‘compromises’ made when different priorities are balanced with respect to what makes a review rigorous.

What this means for the rigour of our evidence syntheses

In theory, having different outputs from the same project should mean that we are able to meet the requirements for rigour as laid out by systematic review collaborations. Having said that, we have found that we do not ‘fit’ in the usual publishing requirements of the systematic review collaborations. For example in 2012/2013 we produced a three-stage review on smallholder farming that included a systematic review of reviews, an evidence map, and a full synthesis. The Campbell Collaboration’s processes were not flexible enough to consider all three steps and only accepted the full synthesis stage, which had to be written up as a standard systematic review, almost as though the first two stages had not taken place [10, 11]. The very fact that our approach has not fit within the usual formats hints that the requirements for rigour within these formats may not be fit for real world decision-makers’ evidence demands.

We are unlikely to be able to employ all the ‘best practices’ promoted by systematic review collaborations as the elements of rigour within our reviews are likely to be broad and responsive to stakeholders’ priorities. Stakeholders’ priorities sometimes take us outside what is considered ‘best practice’. When trying to be responsive to decision-makers’ needs we often have to be quick, which can mean that some steps required for technical/scientific rigour are adapted to the demands of the specific context. For example, having a percentage of papers double screened or double coded rather than all of them, or doing a shorter, less comprehensive critical appraisal stage. The stakeholders with whom we are working may also have priorities for synthesis that do not match those of other stakeholders. This may mean that the review may be of considerable value to some people but not others. This might be for a number of reasons including the fact that the subject that they choose is relevant only to specific environments, or that their outcome of preference does not apply to others’ contexts.

Full publication of all our reviews’ outputs is less likely to take place when we adopt stakeholders’ priorities. Outputs are fed into decision-making cycles immediately without waiting for formal publication processes, which will not necessarily take place. If confidential documents have been included, as has been the case in some of the syntheses we have conducted for government colleagues, it may limit the scope for full publication of data. Quality assurance processes of reviews, such as peer-review, can also look different. Rather than having formal methodological peer-review, decision-makers’ quality assurance processes (and thus definitions of rigour) have to be followed. These can often be different (for example validation meetings of the usefulness of evidence mapping methodologies by a range of government departments) but are not necessarily less stringent: for instance when evidence syntheses are tabled at Cabinet level, the level of scrutiny of the synthesis can be much higher than in traditional academic review as the stakes are higher.

What this means for the relevance and usefulness of our evidence syntheses

We propose that this approach to stakeholder engagement for demand-led reviews is much more likely to be relevant to the needs of those specific stakeholders involved. In our experience of working with the government in South Africa to produce an evidence map on Human Settlements, we developed a conceptual framework that fit closely with the country’s National Development Plan and Mid-Term Strategic Framework. This enabled the evidence map to feed directly into policy debates in government.

Of course such close consultation does not necessarily mean that the synthesis will meet the priorities of other groups of potential users, but we believe that this approach creates more legitimacy as the syntheses are easily recognised as having responded to the priorities and values of the users [7]. Timeliness is such an important factor for decision-makers so by working with them and to their timelines, the review is much more likely to be used: if you miss the policy-window, then the review simply will not be read.

Demand-led reviews move the review design and conduct much closer to the user of the review. This approach changes the balance of power between the researcher and the review user, which can elicit worries about the independence of the review process and findings. Review stakeholders might for example influence the review in such a way as to arrive at the preferred findings and recommendations. In our experience, there are three points to consider in this regard. First, while being more flexible and tailored to decision-making needs, demand-led reviews cannot compromise on the underlying systematic review principles of transparency and following a structured, systematic review approach. Any demand-led review has to comply with these principles as traditional, supply-led reviews do. Second, where vested interests become a challenge to a demand-led review, the review project should be discontinued. However, it is not clear why an independent but unused review is any less a waste of research than a review that cannot be completed due to undue attempts of stakeholder influence. The risk of vested interest due to stakeholder engagement therefore does not seem to present an inherent reason not to conduct demand-led reviews. One could also argue that to challenge and change vested interests and beliefs, if possible at all, engaging with such actors and groups in the review process has a higher likelihood of success than assuming that review findings will reach such groups by themselves. Third, we are not arguing that linking the concept of rigour closely to the review methodology followed is per se not valid. Rather, we are aiming to extend the concept of rigour to not only include methodological soundness, but also questions of the review’s relevance to decision-making contexts, and the perceived legitimacy of the review by the user audience. In this extended definition of rigour then, different aspects can be balanced against each other. However, a review of high-relevance and legitimacy which has achieved these attributes through allowing stakeholders to influence and undermine the review research process certainly would not be considered a rigorous systematic review.

How this relates to approaches taken by others

Our attempts to tailor evidence synthesis methodology to better meet users’ demands and needs have not been developed in isolation. Oliver and Dickinson, for example, highlight the challenges of producing policy-relevant reviews with issues of context and questions about transferability raised [8]. As highlighted in their paper there are issues in relation to translating the global reviews to specific contexts and needs, suggesting that even when these public goods are produced there is considerable translation required to achieve policy-relevance in specific contexts. Some efforts start with this challenge, taking ‘public good’ reviews and aiming to make them more accessible and more likely to be used by decision-makers [12]. This supply-driven approach is different from, but not necessarily contradictory to, our approach.

Others aim to provide evidence response services that are limited in their generalisability and future value, but maximise the potential for evidence-use by decision-makers by meeting their urgent needs [13]. Whilst this meets requirements for rigour in terms of relevance, timeliness, and legitimacy it does not conform to the methodological requirements of full ‘public good’ reviews. The formal systematic review collaborations are shifting slightly in this regard: the Collaboration for Environmental Evidence is discussing where different review products fit, whilst recognising that its primary goal is to produce ‘public good’ systematic reviews; there are indications that the Campbell Collaboration will accept evidence maps in the future.

The greater the number of funders commissioning reviews and the more people from different disciplines apply the method, the more these issues come to the fore and need to be discussed. A good example of this is the recent introduction of evidence synthesis in the humanitarian sector, which motivated a range of interesting debates on rigour and policy-relevance too [14]. We anticipate that there will therefore be more people asking what it means to produce demand-led reviews that respond to stakeholders’ needs, and seeking new approaches such as the one we propose.

What this means for definitions of rigour and what is a ‘gold standard’ review

We set out to discuss a tension that is inherent within the promotion of stakeholder involvement in systematic reviews but is rarely recognised—that to be responsive to stakeholders in producing demand-led reviews requires a re-thinking of what constitutes rigour. This issue is often presented as a tension between rapid evidence assessments and full reviews, but we believe it is a bigger question about what makes a ‘gold standard’ review. We propose that a shift in language is required. We prefer ‘responsive reviews’ to ‘rapid reviews’. We also believe that responsive reviews are not ‘quick and dirty’ but rather ‘quick and good enough’ [15].

We are proposing that a shift in our whole approach is needed, whilst also recognising that this is not always feasible. We believe that responsive reviews remain an important way to increase the use of systematically reviewed evidence in decision-making. At the same time, the inherent value of ‘public good’ reviews for future decision-making remains. We also acknowledge that funding sometimes requires that responsive reviews are done without time for a linked ‘public good’ full review. Perhaps most importantly ‘gold standard’ reviews are not only those that are technically methodologically ‘rigorous’, but are also those that are responsive to decision-makers’ needs and are recognised as being so.

Concluding statement

This commentary aims to address head on the often undiscussed key challenge with regard to stakeholder involvement in systematic reviews: that responding to stakeholders can mean reconsidering what makes a review rigorous. It proposes a new model to address these tensions that combines the production of ‘public good’ reviews with stakeholder-driven syntheses. During 2017, we will be putting this model to the test on a synthesis project exploring ecosystems services' interventions for poverty alleviation in Africa and are looking forward to reporting back on our experience.