Table 7 Study quality assessment framework for experimental field studies
Bias area | Characteristic | Low risk of bias | High risk of bias |
|---|---|---|---|
Study design and sampling | Study type | Before-After-Control-Impact design | Before-After or Control-Impact design |
Treatment replication | Treatment is applied or studied at multiple independent sites | Treatment not replicated at multiple sites | |
Study timeframe | Sampling occurs over multiple years | Sampling occurs only over a single season or year | |
Pairing of nutrient and chl-a measurements | Nutrient and chl-a measurements are taken at the same place and time or index period | Nutrient and chl-a measurements are not paired in time and space | |
Control matching and performance bias | Control and treatment systems are similar and there is a clear effort to account for any differences | Evidence that control and treatment systems differ in aspects not related to treatments, with no effort to account for differences | |
Treatment clarity and detail | Method of nutrient addition clearly explained (e.g. pulse, continuous drip) | Method of nutrient addition not clear | |
Confounding factors | Sample design minimizes effect of obvious confounding factors; or if not controlled by sample design, they are measured and adjusted for in statistical analysis | Confounding factors reported and not accounted for, or are likely, and are not able to be adjusted for post hoc | |
Temporal replication | >1 replicate before and after treatment | 1 replicate before and after treatment | |
Spatial replication | >1 site per control/impact | 1 site per control/impact | |
Measurement clarity and detail | Methods for design and sampling described in detail, including chl-a extraction and measurement, water filtering, and nutrient measurement | Missing information not allowing for repeatability | |
Data analysis and results | Clarity and detail | Analysis methods described in detail sufficient to permit repeating | Missing information not allowing for repeatability |
Uncertainty | Some estimate of uncertainty in effect or relationships provided (e.g. confidence intervals, standard error, standard deviation, etc.) | No estimates of uncertainty provided | |
Treatment vs. Control | Differences in treatments vs. controls reported quantitatively (e.g. actual values, response ratios, effect size) | Differences in treatments vs. controls reported only qualitatively (higher, lower) or are otherwise unclear (e.g. only P value given) | |
Reporting bias | All variables, measurements, and statistical tests mentioned in methods are reported in results or additional file | Some variables, measurements, or statistical tests mentioned in methods are not reported | |
Other biases | Detection bias | No indication that outcomes were measured differently in control and treatment samples | Some indication that outcomes were measured differently in control and treatment samples |
Attrition bias | No differences in loss of control and treatment samples | Differences in loss of control and treatment samples | |
Research aim consistency | Questions clearly stated and answers match questions | Questions not clearly stated or answers do not match stated questions | |
Other bias | No evidence of other sources of bias | Evidence of bias from a source not considered above |