Table 1 Scoring sheet for quality assessment at study level (adapted from Schindler et al. [62])
Bias and generic data quality features | Specific data quality features | Design of assessed study | Quality score |
|---|---|---|---|
Selection and performance bias: study design | Temporal repetition | Complete before–after (BA) time series (>1 replicates before and after) | 25 |
Interrupted BA time series (>1 replicates before and after) | 20 | ||
BA comparison (1 before, >1 after) | 15 | ||
BA comparison (>1 before, 1 after) | 14 | ||
BA comparison (1 before, 1 after) | 10 | ||
Deficient BA comparison (e.g. before-data from archives or not from exactly the same sites) | 2 | ||
No BA comparison | 0 | ||
Spatial repetition | Gradient of intervention intensity including “zero-control”-sites | 25 | |
Site comparison/control-impact (CI) (>1 replicates control and impact) | 20 | ||
Site comparison/control-impact (CI) (1 control, >1 impact) | 15 | ||
Site comparison/control-impact (CI) (>1 control, 1 impact) | 14 | ||
Site comparison/control-impact (CI) (1 control, 1 impact) | 10 | ||
Gradient of intervention intensity without “zero-control”-sites | 5 | ||
Deficient CI comparison (e.g. control-data from archives or not from the same period) | 1 | ||
No CI comparison | 0 | ||
Assessment bias: measurement of outcome | Replicates per treatment (number of sites) | Well replicated (>9 replications) and measured at independent sites | 20 |
Well replicated (>9 replications) and measured at neighbouring sites | 15 | ||
Replicated (4–9 replications) and measured at independent sites | 10 | ||
Replicated (4–9 replications) and measured at neighbouring sites | 8 | ||
Poorly replicated (2–3 replications) and measured at independent sites | 4 | ||
Poorly replicated (2–3 replications) and measured at neighbouring sites | 3 | ||
Unreplicated observations of objective parameters | 1 | ||
Data gathered by expert opinion or questionnaire | 0 | ||
Method to measure outcome | Method to measure outcome perfectly appropriate for purpose | 4 | |
Method to measure outcome of limited suitability | 0 | ||
Coverage | Large scale (large plots, or large overall extent) | 2 | |
Intermediate scale | 1 | ||
Small scale | 0 | ||
Selection and performance bias: baseline comparison (heterogeneity between treatment and control arms with respect to defined confounding factors before treatment) | Sampling | Treatment and control arms homogenous | 2 |
Treatment and control arms hardly comparable due to different sampling OR insufficient information | 0 | ||
Habitat type | Treatment and control arms homogenous | 2 | |
Treatment and control arms hardly comparable due to different habitat OR insufficient information | 0 | ||
Other confounding environmental factors | Treatment and control arms homogenous | 2 | |
Treatment and control arms hardly comparable with respect to confounding factors OR insufficient information | 0 | ||
Selection and performance bias: intra treatment variation (heterogeneity within both treatment and control arms with respect to confounding factors) | Intervention type and intensity | No heterogeneity within treatment and control arms | 2 |
Replicates within treatment and control arms hardly comparable | 0 | ||
Habitat type | No heterogeneity within treatment and control arms | 2 | |
Replicates within treatment and control arms hardly comparable | 0 | ||
Bias linked to clarity and publication bias | Overall consistency and clarity of the paper | High | 2 |
Low | 0 | ||
Statistical approaches appropriate | Yes | 4 | |
No | 0 | ||
Clarity of the description of the method incl. statistical models used | High | 2 | |
Low | 0 | ||
Clarity of the presentation of the results (incl. statistics) | High | 2 | |
Low | 0 | ||
Missing values for nonsignificant results causing publication/reporting bias | No | 4 | |
Yes | 0 |