Skip to main content

Table 7 Study quality assessment framework for experimental field studies

From: Response of chlorophyll a to total nitrogen and total phosphorus concentrations in lotic ecosystems: a systematic review protocol

Bias area Characteristic Low risk of bias High risk of bias
Study design and sampling Study type Before-After-Control-Impact design Before-After or Control-Impact design
  Treatment replication Treatment is applied or studied at multiple independent sites Treatment not replicated at multiple sites
  Study timeframe Sampling occurs over multiple years Sampling occurs only over a single season or year
  Pairing of nutrient and chl-a measurements Nutrient and chl-a measurements are taken at the same place and time or index period Nutrient and chl-a measurements are not paired in time and space
  Control matching and performance bias Control and treatment systems are similar and there is a clear effort to account for any differences Evidence that control and treatment systems differ in aspects not related to treatments, with no effort to account for differences
  Treatment clarity and detail Method of nutrient addition clearly explained (e.g. pulse, continuous drip) Method of nutrient addition not clear
  Confounding factors Sample design minimizes effect of obvious confounding factors; or if not controlled by sample design, they are measured and adjusted for in statistical analysis Confounding factors reported and not accounted for, or are likely, and are not able to be adjusted for post hoc
  Temporal replication >1 replicate before and after treatment 1 replicate before and after treatment
  Spatial replication >1 site per control/impact 1 site per control/impact
  Measurement clarity and detail Methods for design and sampling described in detail, including chl-a extraction and measurement, water filtering, and nutrient measurement Missing information not allowing for repeatability
Data analysis and results Clarity and detail Analysis methods described in detail sufficient to permit repeating Missing information not allowing for repeatability
  Uncertainty Some estimate of uncertainty in effect or relationships provided (e.g. confidence intervals, standard error, standard deviation, etc.) No estimates of uncertainty provided
  Treatment vs. Control Differences in treatments vs. controls reported quantitatively (e.g. actual values, response ratios, effect size) Differences in treatments vs. controls reported only qualitatively (higher, lower) or are otherwise unclear (e.g. only P value given)
  Reporting bias All variables, measurements, and statistical tests mentioned in methods are reported in results or additional file Some variables, measurements, or statistical tests mentioned in methods are not reported
Other biases Detection bias No indication that outcomes were measured differently in control and treatment samples Some indication that outcomes were measured differently in control and treatment samples
  Attrition bias No differences in loss of control and treatment samples Differences in loss of control and treatment samples
  Research aim consistency Questions clearly stated and answers match questions Questions not clearly stated or answers do not match stated questions
  Other bias No evidence of other sources of bias Evidence of bias from a source not considered above