Skip to main content

Table 3 Critical appraisal criteria used to assess included studies

From: What is the impact of human wastewater biosolids (sewage sludge) application on long-term soil carbon sequestration rates? A systematic review protocol

Group Criteriona Low susceptibility to biasb High Susceptibility to Bias
Study design Study type Before-After Control-Impact and Randomized Controlled Trials (though we do not expect to find RCTs as they are uncommon in this field) Control-Impact study designs (no pre-impact data)
Experimental randomizationc Suitable experimental designs for randomization (e.g., completely randomized, randomized complete block, Latin square, factorial, split-plot, strip-plot) Purposive (selective)
Experimental replication (spatial) Replication at level of intervention (i.e., spatial replication) and of large sample size (3 +) Replication at level of intervention but of low sample size (< 3)
Similar starting point Experimental treatment and control groups are similar at the start of the trial Experimental treatment and control groups are not similar at the start of the trial
Randomization Some degree of randomization in sample selection No randomization (i.e., purposive sampling)
Replication of sampling Replication of samples (3+) Low replication (< 3)
Study measurement and data analysis methods Time between intervention (i.e., last amendment/biosolids application and soil measurement sampling) At least 1 year since last application  < 1 year since last application
Soil organic carbon/matter measurement method and details Uses dry combustion/elemental analysis or Walkley–Black titration method to measure soil organic carbon/matter. Uses and details acid treatment to remove inorganic carbonates, if necessary Uses loss-on-ignition or other method to measure soil organic carbon/matter or missing methodological detail to determine soil organic carbon/matter
Soil measurement depth (if incorporated or injected in soil subsurface) Soil measured to at least depth of biosolids incorporation/injection Soil not measured to depth of biosolids incorporation/injection or does not include details to determine depth measured
Soil bulk density measured Soil bulk density measured Soil bulk density not measured
Incomplete/Missing outcome data No missing data; reasons for missing data not related to outcome; missing data balanced across control and intervention groups (and reasons similar); or proportion missing/plausible effect size not enough to have a relevant effect Reasons related to outcome, and imbalance in numbers or reasons; or proportion missing/plausible effect size enough to have a relevant effect
Account for confounding variables Presence of confounders No obvious confounders or adequately accounted for as a result of blocking/pairing Confounders present and/or unaccounted for (e.g., different irrigation strategies with no blocking of added treatment effect)
  1. a Criterion to assess internal validity (risk of bias)
  2. b Unclear classification given to any study where substantial details within the methods are either unclear or missing. Not applicable (N/A) is given to any study where the variable is not applicable to the study
  3. c See Singh and Masuku [24] for appropriate experimental designs and statistical techniques