Table 2 Criteria for study validity assessment
Question/criterion | Response to question | Type of bias addressed | |||
|---|---|---|---|---|---|
Yes | Partially | No | Unclear | ||
1. Did the study consist of both temporal and spatial comparisons? | BACI, RCT | BA, CI, ALT-CI, CI-gradient, RCA, NR | N/A as study is not eligible for inclusion based on inclusion criteria | Lacking sufficient information to judge | Selection |
2. Are experimental/observational units replicated? | ≥ 2 independent experimental/observational units (i.e., the level of replication at which the intervention was administered/the exposure experienced) | There were at least two experimental/observational units but there is a lack of independence between these units (pseudoreplication) | No replication (i.e., < 2 independent experimental/observational units) | Lacking sufficient information to judge | Selection |
3. Are intervention and comparator sites well-matched at site selection/study initiation? | Intervention and comparator sites are well-matched (i.e., similar physical characteristics). N/A for BA designs (i.e., intervention and comparator at the same site) | Intervention and comparator sites are moderately matched. N/A for BA designs | Intervention and comparator sites are poorly matched. N/A for BA designs | Lacking sufficient information to judge | Selection |
4. Can the intervention be clearly interpreted? | It is clear that a change to flow magnitude has occurred and quantitative data on magnitude is reported | It is clear that a change to flow magnitude has occurred but either no quantitative data on magnitude is reported, or the quantitative data is difficult to interpret (e.g., averaged across intervention and control sites within the same river) | N/A | The study compares an unregulated stream (or section of a stream) to a regulated stream (i.e., regulated via a hydro dam), or reports unspecified multiple components affecting flow (i.e., study does not report effects of components separately to isolate individual impacts of components) | Selection, performance, reporting |
5. Was the study free of other potential confounders after sample selection/study initiation? | No or minimal confounding factors present, including e.g.: (1) no or minimal differences in environmental conditions between intervention and comparator sites and/or time periods (e.g., unplanned human alterations, floods, droughts, time-related trends), (2) no additional experimental manipulations of other flow regime components (e.g., flow frequency, duration) at the same time as magnitude alternations. Or if present, are accounted for appropriately in analysis | N/A | Confounding factors present that could have an impact on the outcome and these are not accounted for in analysis | Lacking sufficient information to judge | Performance |
6. Did the study use different sampling/measurement method(s) between intervention and comparators? | Similar/consistent sampling/measurement methods are used between intervention and comparator sites and/or times (e.g., gear type, timing or size of sample areas) | Different/inconsistent sampling/measurement methods are used between intervention and comparator sites and/or times (e.g., gear type, timing or size of sample areas) | N/A | Lacking sufficient information to judge | Detection |