Table 4 Possible ways of classifying risks of bias to inform data synthesis

Low risk/ high risk/unclear risk

This approach has been used extensively in the original Cochrane risk of bias tool for randomised controlled trials [74, 79] and has the advantage that the results are easy to tabulate or present graphically (e.g. using a “traffic light approach” in which red = high, green = low; amber = unclear). However, a disadvantage of having an “unclear” category is that it may be tempting for reviewers to be less decisive and assign most studies to this category

Definitely low risk/probably low risk/probably high risk/definitely high risk

This approach has been proposed by OHAT [80] as a way to avoid having an “unclear” or “no information” category by requiring that instances of insufficient information are recorded within the “probably high risk” category”. Note that OHAT provides explicit criteria defining each category in their guidance [80]

Low risk/moderate risk/serious risk/critical risk/no information

This approach has been used in the latest version of the Cochrane risk of bias tool for randomised controlled trials, with explicit definitions of each category [81]. The categories of “low” and “moderate” risk of bias in Cochrane’s classification are intended to be specifically interpreted in relation to how well the study in question matches an ideal target study design. We also note that several published environmental management systematic reviews included a “medium” risk category (Additional file 2). Where authors use categories such as “medium”, “moderate”, “critical” or “serious”, these should be clearly defined to minimise discrepancies in interpretation