Skip to main content

Table 1 Key problems relating to the application of PRISMA [24] to evidence synthesis in conservation and environmental management

From: ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps

Problem 1:

Does not strictly require a protocol, refers to registration (e.g. PRISMA [24] checklist #5 “Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number”)

Problem 2:

Vagueness hides necessary level of transparency to allow repeatability (e.g. no requirements to provide details of which databases are accessed via Web of Knowledge or Proquest, PRISMA [24] checklist #7)

Problem 3:

Suggested requirements for review conduct are minimal, affecting overall comprehensiveness of the review (e.g. repeatable reporting of search in “at least one database” required only, PRISMA [24] checklist #8)

Problem 4:

Heavy emphasis on meta-analysis (excludes narrative, qualitative and mixed synthesis methods): e.g. PRISMA [24] checklist #13 ‘Summary measures’: “State the principal summary measures (e.g., risk ratio, difference in means)”, PRISMA [24] checklist #14 ‘Synthesis of results’: “Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I2) for each meta-analysis”, PRISMA [24] checklist #21 ‘Synthesis of results’: “Present results of each meta-analysis done, including confidence intervals and measures of consistency”

Problem 5:

Lost nuance of necessary methodological steps: e.g. lack of consistency and comprehensiveness checking, no requirement to avoid bias caused by reviewing and appraising one’s own research

Problem 6:

Focuses on risk of bias rather than limitations to validity: ignores external validity and limitations that do not affect bias (e.g. PRISMA [24] checklist #15 and #22)

Problem 7:

Focuses on medicine and health topics (e.g. PRISMA [24] checklist #9 ‘Results of individual studies’: “For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot”)

Problem 8:

Non-matching terminology: e.g. separating ‘screening’ from ‘eligibility’ (see PRISMA [24] flow-diagram)

Problem 9:

Inappropriate use of the term ‘qualitative synthesis’ meaning ‘narrative synthesis’ (see PRISMA Statement [24])

Problem 10:

PRISMA checklist misses vital information from the PRISMA [24] flow-diagram: e.g. exclusions during critical appraisal

Problem 11:

Often misused as methodological rather than reporting guidance [34]

Problem 12:

PRISMA checklist is only useful for authors and editors during submission: no further information provided within the process of completion: it is purely a checklist