Table 6 Study quality assessment framework for experimental mesocosm studies
Bias area | Characteristic | Low risk of bias | High risk of bias |
|---|---|---|---|
Study design and sampling | Study timeframe | Study timeframe considers risk of container effects (<30 days, but can depend on size and flow-through vs. recirculating) | Study timeframe is long enough to be at risk of container effects and other random processes (typically >30 days but depends on size and flow-through vs. recirculating) |
Randomization of sampling (selection bias) | Some form of randomized assignment of control and treatment samples | No randomization of control and treatment assignments | |
Control matching and performance bias | Control and treatment mesocosms are similar, or there is a clear effort to account for any differences | Evidence that control and treatment mesocosms differ in aspects not related to treatments, with no effort to account for differences | |
Treatment clarity and detail | Method of nutrient addition clearly explained (e.g. pulse, continuous drip) | Method of nutrient addition not clear | |
Confounding factors | No obvious confounding factors reported, or if reported are accounted for | Confounding factors reported and not accounted for, unclear how accounted for, or are likely | |
Replication | ≥3 replicates per treatment, no pseudoreplication | <3 replicates per treatment, or evidence of pseudoreplication | |
Measurement clarity and detail | Methods for design and sampling described in detail, including chl-a extraction and measurement, water filtering, and nutrient measurement | Missing information not allowing for repeatability | |
Data analysis and results | Clarity and detail | Analysis methods described in detail sufficient to permit repeating | Missing information not allowing for repeatability |
Uncertainty | Some estimate of uncertainty in effect or relationships provided (e.g. confidence intervals, standard error, standard deviation, etc.) | No estimates of uncertainty provided | |
Treatment vs. control | Differences in treatments vs. controls reported quantitatively (e.g. actual values, response ratios, effect size) | Differences in treatments vs. controls reported only qualitatively (higher, lower) or otherwise unclear (e.g. only P value given) | |
Reporting bias | All variables, measurements, and statistical tests mentioned in methods are reported in results or additional file | Some variables, measurements, or statistical tests mentioned in methods are not reported | |
Other biases | Detection bias | No indication that outcomes were measured differently in control and treatment samples | Some indication that outcomes were measured differently in control and treatment samples |
Attrition bias | No differences in loss of control and treatment samples | Differences in loss of control and treatment samples | |
Research aim consistency | Questions clearly stated and answers match questions | Questions not clearly stated or answers do not match stated questions |