Updating and amending systematic reviews and systematic maps in environmental management

Abstract

Systematic reviews and systematic maps aim to provide an overview of the best available evidence to inform research, policy and practice. However, like any form of review, they will require updating periodically to ensure that the most recent evidence has been incorporated. Here we outline two types of review revisions as recognised in medicine: updates and amendments. Updates involve a search for new studies, expanding the evidence base through time. Any other change (e.g. in screening or synthesis) or correction to the original report is an amendment. Decisions as to whether/when it is appropriate to undertake an update or amendment must be made on a case-by-case basis, considering issues such as the reliability and scope of the existing review or map, likely volume of new evidence, resources available, and the likely value of including new information. Careful, consistent reporting is necessary to ensure transparency and repeatability, particularly where there are deviations from the original methods, and authors should highlight key advances relative to the original report. Updating environmental systematic reviews and maps will be an increasingly important activity as the numbers of both primary studies and synthetic reports in the literature continue to grow.

Background

Collaboration for Environmental Evidence (CEE) systematic reviews and systematic maps aim to provide an overview of the best available scientific research to help inform research, policy and practice [1, 2]. Since the rate of publication of scientific research is increasing (e.g. Fig. 1), any review is only up-to-date at the point at which the search is conducted [3] and, as with any piece of research, given the time required to undertake and publish a synthesis, there may be a considerable period between completion of searches and production of the final report. As more primary research becomes available, out-dated reviews become less reliable [4]. Some syntheses may not date that quickly (there may be no new evidence generated and made available for years or even decades) whilst others may become out-of-date before they even reach publication. For example, a recent systematic map on the effects of agricultural management on soil organic carbon stocks [5] found increasing publication rates (see Fig. 1) and a full systematic review on one of the sub-topics [6] identified a 23 % increase in the evidence base over a period of just 2 years (unpublished data).

Fig. 1

The number of relevant studies retrieved by year for five recent CEE systematic reviews and systematic maps. Across the five publications, data suggest near exponential growth in the volume of environmental literature

Updates to traditional, narrative-style reviews are not uncommon [e.g. 7–9], but since these reviews lack formally peer-reviewed and published protocols, these works lack the traceability of systematic reviews. Systematic review methods, therefore, offer a high degree of transparency and verifiability where an update is needed. Given the relatively recent introduction of systematic review methodology into environmental management [10], and the even newer introduction of systematic mapping [11], there have been few efforts to bring up-to-date previously completed syntheses. Two update protocols have recently been published in Environmental Evidence and registered with the CEE [12, 13]. These protocols both aim to update existing reviews by the same respective sets of authors and represent the first official updates of systematic reviews or maps endorsed by the CEE.

As with many other aspects of evidence synthesis methodology, medical systematic review frameworks are more developed than other fields with respect to updates and amendments. The Cochrane Handbook for Systematic Reviews of Interventions in medicine proposes that updates involve a search for new studies to expand the evidence base through time, whilst any other change (for example, a change in methods, a correction to the original report or the inclusion of additional evidence without an updated search) is classed as an amendment [14]. We propose that the same distinction is adopted for CEE systematic reviews and systematic maps and we highlight these different approaches below.

Updates

Conducting an update of a systematic review or a systematic map requires repeating the methods detailed in the original protocol, as used to produce the original report, with the aim of capturing and incorporating any information published or produced since the original searches were completed. An update is likely to take less time to complete than a novel synthesis because the question and methods have already been developed, there is a shorter time period over which evidence is sought, and there may be a smaller volume of studies to consider. As an update does not involve any deviations from the methods used previously, a new protocol would not be necessary in most cases. Updates are likely to be particularly valuable in enabling systematic reviews or maps to be kept up-to-date through regular re-runs of the searches and incorporation of new studies into the review or map database. However, updates to systematic reviews and systematic maps that re-run the original searches (assuming the use of precisely the same inclusion criteria) but fail to find any additional studies for inclusion are also valuable in showing that the original report is up-to-date at the point of the current searches. Updates would require that all methods used to search for, screen, extract data from and synthesise studies remain identical to the methods specified in the protocol and used for the original review. Any planned deviations whatsoever would require an amendment.

Amendments

Amendments are systematic reviews or systematic maps that involve a change in the materials and methods of the review (i.e. the protocol). An amendment requires transparent documentation and justification, along with peer-review by subject and methodological experts, and necessitates the submission of a new protocol and full report that describe any deviations from the original methods. Most updates to existing CEE systematic reviews are likely to need to be undertaken as amendments. This may be because key elements of the review topic have changed (such as novel interventions, primary research methods, or terminology), or since methodology is advancing rapidly and novel approaches to synthesis are continuing to be developed [e.g. 11]. Since the CEE guidelines are being continually developed and revised to help improve the standards of syntheses, older systematic reviews and maps may not meet best practice levels of methodological rigour, and so amendments provide an opportunity to address any limitations of the previous methods used [4]. For example, aspects of critical appraisal may be improved upon in an update where previous methods were not comprehensive or were absent. Amendments also offer an opportunity to improve the reporting of the synthesis to increase transparency [15] or to correct errors in the materials and methods of the original review [14]. An amendment would also be appropriate if an existing systematic review or map was considered to need revising methodologically (e.g. if the meta-analytical or critical appraisal approach used was disputed or could be substantially improved) despite being otherwise up-to-date.

Reviewers may wish to include research in a language not previously included in a published systematic review or map. This would require changes to the methods (including the search strategy) and, as such, an amendment would be necessary.

Review team composition

Current CEE policy is to initially offer the opportunity to update a systematic review or systematic map to the original authors [4]. However, updates or amendments may be undertaken by a review team containing a mixture of new and original reviewers, or an entirely new review team may be formed. Each scenario offers advantages. Authors of the original reports would be familiar with the topic and the sources searched, may be in touch with any stakeholders that contributed data or expertise, and may have access to original files. This can minimise duplication of effort by allowing, for example, automatic removal of previously screened records from the updated search results. New reviewers may be in a position to critically assess limitations of the original review methods, provide additional complementary expertise or different interpretation of the results, and spot human errors in the previous review methods. Having a mixed review team is also beneficial in maintaining consistency with the original systematic review or systematic map, for example by ensuring that inclusion criteria are applied consistently (although updated checks of consistency should still be undertaken to assess consistency over time). The CEE has stated that the review team must have the agreement of all authors before submitting a protocol or full report to Environmental Evidence [4].

Deciding when to update or amend

There is much uncertainty as to when and how evidence syntheses should be updated or amended. Most of the focus on this topic in the literature has been on medical and health-related systematic reviews [e.g. 15–19]. One study [16] found that, of 362 reviews published in the Cochrane Database of Systematic Reviews in Issue 2 (published in 1998), 4 years later 38 % had been updated or amended to include new studies, 32 % had re-run searches but no new studies were included, and 30 % had not been updated or amended. Of the updated or amended reviews, 9 % had a change in reviewer conclusions [16]. Yet even when the conclusions do not change there can be other valuable implications for the review, including increased reliability due to the inclusion of novel evidence and the ability to demonstrate comprehensiveness and continued relevance. For example, updating and amending a systematic review on nicotine replacement therapy gave a more precise summary effect estimate although the clinical message was unchanged, and allowed data addressing additional questions to be included in the new review [19]. In environmental sciences, the effect of including novel research in a synthesis will depend not only on the size of the evidence base but also the reliability of new evidence relative to that within the existing synthesis. Meta-analysis allows reviewers to estimate what kinds of new data would be needed to significantly overturn the results, and these types of analyses could help to consider whether an updated or amended review is necessary.

In medicine, despite recognition that updating and amending evidence syntheses is important to ensure the most accurate and up-to-date evidence is available, such practices are not common and many organisations that commission or produce systematic reviews do not have formal policies in place [17]. Furthermore, little research has been undertaken into when and how best to update or amend systematic reviews and so a range of strategies, methods and models exist [18, 20]. The Cochrane Handbook of Systematic Reviews of Interventions advocates that reviewers should run searches every 2 years to determine whether a novel review is necessary [14], whilst current CEE guidance for environmental reviews is to update or amend every 5 years [21]. A quick scoping exercise (searching and/or contacting experts) can help to identify when a new review may be necessary or useful. In addition, reviewers considering an update or amendment can examine the number of included articles published over time in the original review to estimate the publication rate. This can allow a rough prediction of the number of new studies likely to be found in recent years (Fig. 1). Whilst influential, the frequency of publication of articles does not necessarily mean that topics with lower publication rates do not warrant updates or amendments, since it is the strength of evidence that is important, not the volume. There is a range of other factors that may influence a decision to update or amend a systematic review or map (Table 1).

Table 1 Factors affecting a decision of whether to conduct an update or amendment to an existing systematic review or map

Conducting an update or amendment

The first task for the review team is to determine the purpose of the review revision. If the aim is to simply conduct an update following the existing methods, the review team should notify CEE of their intent before beginning to update the searches. Updating the searches will simply involve re-running them as per the original review or map. Update searches can often be filtered by publication year: a small overlap with the original search is recommended to allow for time lags in the indexing of studies and for searches that covered only part of a calendar year.

If any aspects of the methods or analysis need to be improved or upgraded (for example, to take into account developments in the CEE guidelines or any recommendations identified by the original reviewers in the ‘Limitations of the review/map’ section) then an amendment is necessary and the review team will need to prepare a new protocol for submission to Environmental Evidence (the CEE dedicated journal) specifying the changes. At this stage, the review team must identify whether there is any additional descriptive information or quantitative data that may yet need to be extracted from the studies that were included in the previous review to allow for extended critical appraisal, advanced meta-analysis or inclusion of additional data. This is particularly important since it may have significant resource implications.

Where there is any uncertainty as to whether it is necessary to conduct an update or an amendment, the review team should consult with CEE at the earliest possible opportunity. The review team should also check the original systematic review or map report to determine what data or information are available to them, and in which formats. For example, if original EndNote files are available it may make removing duplicates from updated searches easier than if references are only presented in tables, as lists or even in hard copy. Depending on the composition of the review team, it may be desirable to assess consistency of screening activities on samples used by the original team (if available) to ensure comparability in the application of the inclusion criteria.

Once the update or amendment is underway, the process is similar to that of conducting a new systematic review or map, and guidance on the process should be sought from the relevant sections of the current CEE guidelines (see http://www.environmentalevidence.org). A report of an update may be relatively brief, especially where no new studies have been identified, whereas an amendment will require a more comprehensive report. As per the Cochrane Handbook for Systematic Reviews of Interventions, a ‘What’s new?’ table could usefully be included to help highlight to readers what has changed [14], or the review team could simply include a bulleted section in the text. Authors should seek to ensure transparency and repeatability by documenting any deviations from the protocol and by providing additional files as supplementary information, and in their native file formats (i.e. the formats used to produce and edit the files) where possible. It is also helpful to use a modified flowchart to show how additional results have been captured and incorporated in with the original results (see Fig. 2).

Fig. 2

Example of a flow diagram for a hypothetical amended systematic review. Figure demonstrates how an updated search has been combined with that of the original systematic review to give the total number of studies included in the updated systematic review

Review authors need to communicate the findings of their updated or amended review or map to ensure that end users are aware that an update has been completed. As such, authors may wish to produce an update brief document for publication alongside their review in Environmental Evidence (perhaps as supplementary information or within the review report itself), for example, that highlights the advances made by the new version and could be communicated to relevant stakeholder communities.

Preparing for an amendment or update to a systematic review or map

Caution should be urged here to ensure that reviewers undertaking an update or amendment have critically appraised the methods used in the original systematic review or systematic map to ensure the methods are still considered to be robust. Authors conducting new CEE systematic reviews and maps may wish to include a section on ‘Future updates or amendments to the review/map’ that details their recommendations in order to facilitate the process of updating or amending. For example, the authors may recognise that their review would have benefitted from the inclusion of additional data or from a different critical appraisal tool, and as such may recommend a future amendment. Authors of a systematic map with a comprehensive search strategy may simply recommend an update to check for and incorporate any future studies in a period of 2 years subject to available resources. The authors may also wish to consider the files and formats that they make available with their systematic review or systematic map report to help facilitate any future updates. Including citation libraries (for example in RIS format) for all search results and at each inclusion stage would allow automatic removal of previously screened results.

Authors of systematic review and map updates and amendments should ensure that they are fully aware of any issues relating to copyright of documentation procured from the original review and used in any novel, published work. All articles published in Environmental Evidence are (at the time of publication) licensed under the Creative Commons Attribution License 4.0 (https://creativecommons.org/licenses/by/4.0), which allows the work to be shared and adapted providing appropriate credit is given by the authors (i.e. clear citation) and all adaptations are stated clearly (i.e. specified in the updated review or map, or in the amendment protocol and review or map report).

Conclusions

This paper outlines a framework of best practice for updating and amending systematic reviews and systematic maps for publication in Environmental Evidence, as endorsed by the CEE. The need for an update or an amendment should be determined on a case-by-case basis. Although syntheses on broad, topical and fast-moving issues may require more frequent updating to capture novel literature, those on narrower topics may require updating less frequently. Whilst influential, the frequency of publication of articles does not necessarily mean that topics with lower publication rates do not warrant updates, since it is the strength of evidence that is important, not the volume. Updates should be relatively rapid to conduct, whereas amendments may require greater time and resources and the production of a new protocol. In practice, updates are likely to be performed much less frequently than amendments, since minor changes to the methods used in original reviews and maps are unavoidable in many cases. Clear reporting and full archiving of data and citation libraries for original systematic reviews and maps would facilitate updates, and current reviewers should seek to maximise transparency by documenting as much of their activities as possible. Careful, consistent reporting is necessary to ensure transparency and future repeatability, particularly where there are deviations from the original methods. Although there are a number of challenges to consider, updating and amending environmental systematic reviews and systematic maps are likely to be increasingly important as the numbers of both primary studies and syntheses in the literature continue to grow.