Will environmental flows increase the abundance of native riparian vegetation on lowland rivers? A systematic review protocol

  • Kimberly A Miller1Email author,

    Affiliated with

    • J Angus Webb1, 2,

      Affiliated with

      • Siobhan C de Little1 and

        Affiliated with

        • Michael J Stewardson1

          Affiliated with

          Environmental EvidenceThe official journal of the Collaboration for Environmental Evidence20121:14

          DOI: 10.1186/2047-2382-1-14

          Received: 27 January 2012

          Accepted: 13 December 2012

          Published: 17 December 2012



          The extraction of water and alteration of flow regimes by humans have profound negative effects on river ecosystems. Returning water as “environmental flows” is a primary method for restoration, but evidence linking flow restoration to ecological benefits is weak. In order to draw more informative conclusions about the effects of environmental flows on ecosystems, reviews of ecological responses to altered flow regime need to focus on relationships between causes (flow components) and effects (ecological responses). We will review the literature on the responses of native riparian vegetation to flow alterations on regulated rivers. This review should improve river restoration efforts by identifying which flow components can be targeted by environmental flows to improve vegetation condition and increase abundance at the individual, population, and community levels.


          We will conduct our review using the Eco Evidence framework, a novel, freely-available systematic review method and software that employs a standardised methodology to assess cause-effect hypotheses in the face of weak evidence. We will search published and grey literature for studies that present primary data on the responses of native riparian vegetation on lowland river banks to changes in flow regime. The review will assess evidence for seven distinct hypotheses that include different flow components (flood area, depth, duration, frequency, seasonality, and volume) and vegetation responses (condition, germination rates, reproduction, and survival). We will extract information from relevant studies on the trajectories of causes and effects, the type of study design, and the number of control and impact sampling units. This information will be used to weight studies, where studies with more sampling units or stronger study designs are given a higher weighting, as spurious results are less likely. The amount of weighted evidence supporting and refuting each hypothesis will determine which of four possible outcomes we will reach: “Support for hypothesis”, “Support for alternative hypothesis”, “Inconsistent evidence”, or “Insufficient evidence”. We will then collectively consider the conclusions for each hypothesis in order to answer our primary question, summarise the evidence, identify any gaps in knowledge, and provide recommendations for whether and how environmental flows could be used in the management of native riparian vegetation.


          Floodplain River Restoration Germination Condition Mortality Reproduction


          The modification of rivers by humans has profound negative effects on river ecosystems, particularly through water extraction and the alteration of flow regimes [1]. A primary method for river restoration is returning water to stressed rivers as “environmental flows” to restore a more natural flow regime (e.g. [2]). Restoration requires the development of general frameworks for predicting ecological responses to flow alterations [3], but scientific evidence for ecological benefits of environmental flows is weak because of a poor record of long-term monitoring and evaluation of existing flow restoration efforts [4]. In an extreme example, no monitoring or evaluation was carried out for ~90% of river restoration projects in the USA worth approximately US $15 billion [5]. Those that did had poor study designs and a lack of rigorous monitoring, resulting in limited data and a weak evidence base upon which to determine the ecological benefits; however, this information is central to the better management of water resources.

          In Australia, water managers are currently overseeing the largest transfer of water from consumptive to environmental uses in the nation’s history. Governments at the state and commonwealth levels are investing billions in water entitlements to be held as environmental water [68]. Demonstrating the ecological benefits of environmental flows has never been more important because of the substantial investment of public funds. However, the widely-accepted hypotheses that underpin environmental flow regimes have not been rigorously tested, and are thus open to challenges. Systematic reviews of the existing evidence in the literature can assess the evidence basis for the predictions of environmental flow assessments.

          Reviews of individual case studies are difficult because of the large number of potential ecological responses (e.g. fish abundance, fish diversity, vegetation condition) and specific flow alterations (e.g. changes in the magnitude, timing, duration, or frequency of floods or baseflows). Previous reviews have documented evidence that flow alteration is associated with ecological change, but many have been inconclusive [912]. Major taxonomic groups may respond differently to flow regimes, and changes to different flow components may have opposing ecological effects [11]. For example, while 61 of 70 studies (87%) showed a relationship between reduced flow volume and various ecological or geomorphological changes, a quantitative analysis of a subset of these studies that provided the magnitude of hydrological change and ecological responses could not reject the null hypothesis that ecological change was independent of changes to flow volume [10]. Similarly, in a more comprehensive narrative synthesis of the literature, Poff and Zimmerman [13], found that 152 of 165 publications (92%) reported a negative ecological response to a variety of flow alterations, but quantitative analyses did not find any robust statistical relationships. Ecological responses were also not consistent across taxonomic groups: metrics of macroinvertebrate and riparian vegetation response both increased and decreased in response to flow alteration; fish abundance, diversity, and demographic rates showed consistently negative responses to changes in flow [13]. In order to draw more informative conclusions about the effects of environmental flows on ecosystems, reviews of the ecological responses to altered flow regime need to focus on relationships between causes (flow components) and effects (ecological responses). This systematic approach can inform environmental flows science by identifying which flow components are most likely to result in ecological benefits if enhanced with environmental flows.

          Intended ecological outcomes of environmental flows are often river- or reach-specific, but general goals can be identified to form appropriate topics for systematic reviews. One such generalised ecological outcome that often influences the design of environmental flow programs is an increase in abundance of native riparian and floodplain vegetation. This objective may be related to a single species (e.g. Wimmera bottlebrush, Callistemon wimmerensis, [14]), dominant groups of species (e.g. Populus fremontii and Salix gooddingii, [15]), or the entire riparian fauna (e.g. Green River, [16]). Riverine flow regimes govern, to a large extent, the composition, structure, and abundance of native riparian vegetation, because morphological, life history, and phenological traits have evolved under the selective pressures of specific flow components (e.g. flood timing, volume, duration, and frequency, and drought; [3, 17]). Extensive modification of flow regimes by humans has altered riparian vegetation communities (e.g. [18]). Environmental flows targeted at the restoration of various components of flow regimes should improve riparian vegetation quality and abundance at the individual, population, and community levels.

          Objectives of the Review

          As part of a large-scale program for the monitoring and evaluation of environmental flows in Victoria, Australia (the Victorian Environmental Flows Monitoring and Assessment Program, VEFMAP; [19, 20]), we have existing partnerships with six agencies for water catchment management [21]. Through a series of questionnaires and interviews, this group of stakeholders identified the ecological outcomes of environmental flows for vegetation considered by stakeholders to be of greatest importance in their catchments. Ecological concerns vary greatly across these regulated rivers, so ecological responses were identified at broad levels that could be applied to all rivers (e.g. species-specific responses were refined to include broader taxonomic or functional groups). The primary question for this review is one of the ecological outcomes of highest priority identified. The secondary questions are the direct cause-effect linkages, and expected trajectories, identified in our conceptual model (Figure 1). Many of these relationships are commonly accepted on the strength of experiential evidence, but a systematic review is required to test the evidence for each hypothesis.
          Figure 1

          Conceptual model linking changes in flow to an increase in the abundance of native riparian vegetation on banks. Where different hypotheses are posed for woody and herbaceous vegetation, the type and life stage is specified. Dotted lines represent assumed linkages. Solid lines represent conceptual linkages to be tested in our review, and form each of the secondary questions. Overbank flows are those of sufficient magnitude to breach the banks of the channel. High flows are of a greater magnitude than baseflows and can be bankfull or overbank. Sustained high flows can refer to multiple flow components (volume, frequency, seasonality, duration) depending on the study system (e.g. an increase in duration of a high flow in a single year or an increase in volume over multiple years).

          Primary question

          Will the delivery of overbank flows and high flows increase the abundance of native riparian vegetation along lowland rivers?

          P: lowland rivers globally

          I: delivery of overbank flows and high flows

          C: no intervention

          O: increased abundance of native riparian vegetation

          Secondary questions

          1. 1)

            Do seasonal overbank flows increase germination rates of native riparian plants?

          2. 2)

            Do seasonal overbank flows decrease survival of woody seedlings and/or saplings and herbaceous riparian vegetation?

          3. 3)

            Do seasonal overbank flows increase survival of adult woody vegetation?

          4. 4)

            Do sustained high flows decrease survival of woody seedlings and/or saplings and herbaceous riparian vegetation?

          5. 5)

            Do more frequent high flows increase survival of woody seedlings and/or saplings and herbaceous riparian vegetation?

          6. 6)

            Do more frequent high flows improve the condition of native riparian vegetation?

          7. 7)

            Does better condition result in increased reproduction of native riparian vegetation?



          Effective management and restoration relies on understanding causal relationships between environmental stressors and ecological responses. However, demonstrating causality in ecology is difficult because of natural variability, lack of replication, the presence of confounding influences, and limitations on experimental manipulation. Facing similar issues, epidemiologists developed ‘causal criteria’ in the 1960s to assess causality in the study of disease. Causal criteria analysis is a standardised methodology developed for assessing cause-effect hypotheses in the face of weak evidence, and is widely used in medical research [22, 23]. When assessing the strength of causal inference from a systematic review, it is implicit that the results should be assessed against the causal criteria [24].

          Our review will be conducted using Eco Evidencea, a freely available method [25] and software [26], which was recently developed to facilitate causal criteria analysis in environmental science, by employing the literature as a source of evidence. It is currently the only method for systematic review in environmental science supported by software. The history and logic behind causal criteria generally, and the Eco Evidence framework specifically, are described in detail elsewhere [25, 27, 28]. Eco Evidence is philosophically aligned with existing methods of systematic review [24], and has several advantages over the methods used for most reviews in ecology, which normally only attempt some sort of narrative summary of the literature [29]. These include: including standard terms and criteria for the evaluation of each study, the ability to separately evaluate each linkage in a conceptual model (thus more specific ecological questions), and an explicit synthesis of the evidence for and against each of these linkages. Consequently, Eco Evidence reviews are more concise and targeted than is commonly the case for ecological reviews [25]. The Eco Evidence method is also supported by software [26]. The software consists of a web database for storing and sharing “evidence items” (information from individual studies) and a desktop analysis tool that uses evidence shared via the web application. A wizard-style interface guides users through an 8-step causal criteria analysis to examine evidence from a sample of relevant studies [25, 27, 28]. It produces a full report at the end of the process, maximising transparency of the assessment. Together, these methods and tools provide complete transparency of the review, and facilitate consistency of results across multiple reviewers and/or long time-frames. Causal criteria analysis does not require quantitative summary statistics, thus allowing for the inclusion of about twice the number of studies than meta-analysis [30]. Because studies that provide evidence of an association between a cause and effect may be overrepresented in meta-analyses [31], it has been suggested that the Eco Evidence approach to study inclusion could reduce publication bias [30]. We will test this hypothesis in our review (see “Data synthesis and presentation”).

          Briefly, the first four steps involve the formulation of the problem, and are documented above: (1) describing the overall question to be assessed (Primary question), (2) identifying the context (Introduction), (3) developing a detailed conceptual model (Figure 1), and (4) identifying relevant cause-effect hypotheses that will be evaluated (Secondary questions). Step 5 comprises the literature search and systematic review using a rule-based approach to weight individual studies. Each study is reviewed for relevant data (termed “evidence items”), which are then given a weighting based on the quality of the study (termed “evidence weight”) according to a standardised set of rules. These criteria are described in detail under “Data extraction strategy”. If necessary, the conceptual model and/or cause-effect linkages are then revised in Step 6. The last two steps are a systematic analysis of the evidence and assessment of the level of support for the overall hypothesis. In the analysis, evidence weights from each study are summed (Step 7), and the total evidence “in favour of” and “against” each of the cause-effect hypotheses (secondary questions) are used to determine whether sufficient evidence exists to support or refute the overall question (primary question; Step 8).

          Eco Evidence has successfully been used in several recent literature reviews concerning aquatic systems [30, 3235]. Three of these reviews examined responses of aquatic and riparian plants to changes in water regime, but ask different questions to those in this protocol (impacts of changes to inundation timing on riparian vegetation processes; [30]; effects of changes in hydrology on wetland plant species; [35]; ability of environmental flows to reduce encroachment of terrestrial vegetation into regulated river channels; [34]). The recommendations of all case-study authors to promote transparency and increase rigour have been incorporated into our protocol.

          Search strategy

          Scope of the search

          Our choice of databases is designed to provide good coverage of both the published and grey literature, whilst also conserving project resources for the actual review [24]. We will search the published literature using ISI Web of Science, SCOPUS, and Expanded Academic ASAP. We will also search two databases for grey literature: Agriculture and Natural Resources (ANR) Index and CSA Illumina (AGRICOLA, ASFA: Aquatic Sciences and Fisheries Abstracts, Biological Sciences, Environment Abstracts, Meteorological & Geoastrophysical Abstracts, Plant Science). The latter two databases will be used only to search grey literature, as they provide flexible advanced search options, allow complex Boolean operators, and export search results. The results returned from these databases would be less biased than spot searches of the internet or searches of selected NGO and GO archives, which require subjective choices about the repositories searched.

          We will conduct separate literature searches for each secondary question identified, so that each link in the conceptual model is investigated separately. Searches will be conducted in English only, because of the high volume of hits returned in preliminary searches [1,800-15,000 for each secondary question]. A preliminary search of Web of Science suggests that 98% of search results are in English.

          Search Terms

          Three limiting term searches will be used to narrow results to studies of flows on riparian vegetation on riverbanks. These will be combined, as appropriate, with search strings for each vegetation response identified in the secondary questions (Table 1).
          Table 1

          Search terms for limiting term searches (TS) and each vegetation response




          vegetation OR tree OR wood* OR herb$ OR herbaceous OR understory OR grass* OR shrub OR forb


          floodplain OR riparian OR river* OR stream OR creek OR bank


          Inundat* OR bankfull OR flow$ OR overbank OR freshes OR flood$ OR “pulse-flood*” OR “flood release*” OR “water regime” OR “water-level” or hydroperio*


          Germina* OR seedling OR sapling OR growth OR recruit* OR establish*


          Mortality OR dieback OR surviv* OR death OR resilience OR drown*


          Condition OR health OR dieback OR “drought stress” OR “flood stress” OR crown


          Reproduc* OR seed* OR propag* OR flower

          Because language used to describe flow type varies widely, it is not possible to search for one flow ‘component’ at a time (e.g. seasonal overbank flows or high flows). Thus, the secondary questions will be simplified into four searches that cover the seven secondary questions (Table 2). The results for each search will be managed in a separate reference library, but literature reviews will separately evaluate each of the seven secondary questions.
          Table 2

          Complex search operators used to search the published and grey literature

          Simplified secondary question

          Boolean search operator

          Do flows increase germination rates?

          TS1 + TS2 + TS3 + Germination

          Do flows increase/decrease survival?

          TS1 + TS2 + TS3 + Mortality

          Do flows increase condition?

          TS1 + TS2 + TS3 + Condition

          Does condition affect reproduction?

          TS1 + TS2 + Condition + Reproduction

          Simplified secondary questions combine all flow types and vegetation responses.

          Study inclusion criteria

          For each search of the simplified secondary questions, we will randomise results and skim the titles and abstracts for relevance. Randomisation of search results and sequential assessment is proposed a priori to avoid bias when a very large number of relevant studies are available (see “Data extraction strategy”). Repeatability of our assessment for relevance (below) will be determined through an evaluation of a subset of the search results by two investigators working independently. A Kappa value of 0.6-0.8 (substantial agreement) will be required for inclusion/exclusion of papers at this stage [36].

          Papers will be considered relevant to our review if they present primary data (i.e. reviews are irrelevant) on the responses of native riparian vegetation on stream/river banks to changes in flow regime. The vegetation response to flow alteration need not be the primary focus of the paper; for example, flood frequency may be a covariate in a study focussed on accessibility to grazing stock.

          Relevant population(s)

          Studies that investigate the response of native riparian vegetation on lowland river/stream banks globally are included in this review, irrespective of scale and climate. Mountain streams and permanent wetlands are not considered relevant populations. The study must not report on a severely threatened species, as environmental flows do not typically act as the sole approach for threatened species conservation, where non-hydrological factors may be the primary cause of decline. Whilst the results will be immediately used in a project on lowland rivers in south-eastern Australia, the evidence may come from lowland riparian ecosystems worldwide, and the results could be generalised to these ecosystems if the results show consistent patterns across studies.

          Relevant intervention(s)

          Where they result in changes in overbank or high flows, studies that investigate changes in six components of flow regime will be considered: flood area, depth, duration, frequency, seasonality, and volume. Two of our secondary questions (numbers 5 and 6) are restricted entirely to studies of the frequency of high flows; one (number 4) is restricted entirely to studies of the duration of high flow. The study can document either an increase or decrease in flow, but must include a detailed description of the change. The interventions that cause a change in flow regime may include river regulation (dams, weirs, hydropower, extraction) or natural changes in flow regime due to drought or flooding. The study will be considered irrelevant if it reports on an extreme event (e.g. 100-year flood), as environmental flows do not typically aim to restore extreme flood events.

          Our last secondary question (number 7) is unrelated to flooding, but relevant interventions will be changes in the condition of riparian vegetation caused by differences in flow regime. Where riparian condition differs because of parasitism, land-use, or other non-hydrological stressors, the studies will be considered irrelevant.

          Relevant comparator(s)

          Any study that investigates a change in flow regime, and provides a detailed description of the change in flow will be considered (e.g. “the frequency of overbank flows decreased from 1 in 2 years to 1 in 20 years”). The comparators can therefore be a greater or lesser flow compared with the intervention. For example, the frequency of overbank flows may be reduced after river regulation (higher flow frequency before the intervention would be the comparator). Alternately, flood durations may be greater in summer on a regulated river compared with an unregulated river (shorter flow duration on the unregulated river would be the comparator).

          Relevant outcome(s)

          Studies will be considered relevant when they outline the responses of any native riparian vegetation on stream/river banks. We will consider the responses of both woody and herbaceous vegetation, and the association may be statistical or subjective in nature. We will include any trajectory (increase, decrease, no change, or non-directional changes) of several attributes of vegetation: germination rates, survival of seedlings/saplings, survival of adults, condition, and reproduction. Growth rates will not be included as measures of condition, nor will abundance and density measures be used as surrogates for any of the attributes. Leaf Area Index, canopy condition, canopy dieback, water potential, percentage of dead branches, and similar measures will be included in condition. Studies of both sexual and vegetative reproduction will be included.

          If identified as relevant, we will extract all relevant evidence items from a paper. In the Eco-Evidence approach, a single paper can contribute evidence items to multiple linkages (e.g. to “overbank flows increase germination rates” and “overbank flows increase survival of woody adults”), but cannot contribute more than one piece of evidence to a single linkage (e.g. “overbank flows increase germination rates in species A” and “overbank flows decrease germination rates in species B”). This rule prevents against "double weighting" of an individual study in the analysis. We will combine evidence for woody and herbaceous vegetation, except where the conceptual model identifies separate hypotheses for their responses.

          Data extraction strategy

          Data from each relevant study will be extracted according to the standardised methodology in Eco-Evidence (Step 5; [25]). We will record the trajectories of causes and effects (either increase, decrease, change, or no change; e.g. “increase in overbank flow, increase in germination”; or “decrease in overbank flow, no change in germination”). The trajectories of the cause and effect determine whether an evidence item provides evidence that supports or refutes the hypothesis. If several species in a study are all relevant to our hypothesis, the trajectory of the response will be determined by the response of the overwhelming majority of species. We will also record the type of study design from a broad list of categories [“Before v. After (no control)”, “Control/Reference v. Impact (no before)”, “Spatial gradient”, “Temporal gradient”, “BACI/ BARI, MBACI/MBARI” or “After impact only”] and the number of independent control and impact sampling units. Assigning the type of study design and the number of sampling units will follow the definitions and guidance of Nichols et al. [27]. Sampling units must be recorded at the scale of the hypothesis being tested in the review [25], and so are sometimes larger than as presented in the study being assessed. ‘River’ or ‘Impoundment’ is a frequent sampling unit for the studies examined to date.

          Reviewing a very large number of relevant studies for a single secondary question will consume extensive resources for a diminishing return. If a very large number of relevant studies is found for the individual secondary questions (see above), we will extract evidence from 25 randomly-selected publications for each linkage, and use re-sampling techniques to test the representativeness of 20-publication subsets from this set (see below). Nichols et al. [27] claimed that Eco Evidence allows clear results regarding the support or refutation of a cause-effect hypothesis to be obtained with a subsample of the relevant literature, rather than a complete review. Our study will formalise this sub-sampling and test its validity, and represents an advance on the previously-published Eco Evidence method. Fundamentally, this sub-sampling approach rests on a basic assumption in research: that a random selection of possible data should be representative of the entire population - in this case, the available knowledge.

          For secondary questions with more than 20 relevant studies, we will assess repeatability of the outcome as follows. Using the full set of relevant studies (up to 25 per secondary question), we will conduct re-sampling analyses (jack-knife or bootstrap) to develop a distribution of 20-study samples and quantify the rate at which summed evidence weights and conclusions change as a result of re-sampling (if ever). We emphasize that we expect that this sub-sampling approach will be used rarely. In practice, it is common that less than 20 relevant studies are found after assessing all the literature for a single secondary question. On the rare occasions when 20 relevant studies are found, it is only once a large proportion of the literature has been assessed. For example, in one completed review, we failed to reach 20 studies for 4 out of 5 hypotheses tested. For the remaining hypothesis, we had to assess 70% of the search results to find 20 relevant studies [34]. The chance of substantial non-representativeness for a sample drawn from this much of the population is extremely low. For the current protocol, an appendix will include an MS Excel file of the full list of titles returned from the literature search, along with the random numbers used to select titles for initial assessment, a record of which were assessed initially as relevant, and which were used in the review.

          Study quality assessment

          With an Eco-Evidence approach, all relevant studies are included, regardless of their quality. The criteria for calculating standardised evidence weights accounts for study quality so that higher-quality studies play a greater role in the assessment [25]. As a part of data extraction, we note the study design type and the number of independent sampling units. The information on study design and replication is used to weight the individual evidence items for analysis, where studies with more sampling units or stronger study designs are given a higher weighting, as spurious results are less likely (Table 3; [25]). Evidence weights for individual studies can range from 1–10 (Table 3). In this review, studies will not be weighted based on geography; our goal is to elucidate general ecological relationships regarding riparian vegetation. If after analysis, the results indicate that different patterns emerge in different geographic locations, it will be possible to refine the primary and/or secondary questions to limit the geographic scope.
          Table 3

          Weights applied to evidence items, based on the study design and number of independent sampling units [ [27]]

          Study design

          Weight applied

          After impact only


          Reference/control vs. impact (no before data)


          Before vs. after (no reference/control data)


          Gradient response (temporal or spatial)


          BACI/BARI or MBACI/MBARI or Beyond MBACI


          Factorial design: Number of independent reference/control sampling units








          Factorial design: Number of independent impact sampling units








          Gradient designs: Number of sampling units










          The evidence weight is the sum of the weight applied for the study design and the number of sampling units. See text for an explanation of weights. "B" = before, "A" = After, "C" = control, "R" = reference, "I" = impact, "M" = multiple.

          Data synthesis and presentation

          Data is synthesised in Eco Evidence according to a set of rules that maximise transparency in the conclusion [27]. These rules were derived from extensive consultation with ecologists and numerous trials (see [25]). To evaluate support for each cause-effect linkage in the conceptual model, Eco Evidence applies a default threshold of 20 points summed from the evidence weights for all studies. The Eco Evidence method states that both weights and thresholds may be changed if justified by the application, but that any such change must be documented (e.g. [32]). Without any such justification, we will use the default threshold, as it has proven useful in previous studies [30]. This threshold means that a few strong evidence items are sufficient to support (or refute) a hypothesis, but many weaker evidence items would be needed to reach the same conclusion. Weaker evidence items are more likely to be confounded by uncontrolled environmental variables than higher-quality evidence items [25]. Therefore individual weak studies are more likely to reach spurious conclusions. However, if different individually-weak studies are conducted in different places, by different researchers, and using different experimental designs and methods, they will not be affected by the same confounding variables. Therefore a diverse collection of weak studies are unlikely to reach the same spurious conclusion. In such circumstances, a diverse collection of weak studies that describes a consistent association between putative cause and effect, provides strong overall evidence for a causal relationship [37]. The use of weaker evidence actually allows for the inclusion of more of the literature, making the most of the scant evidence available [25]. We will include the mean and range of evidence weights and the types of study designs used to inform the conclusion, allowing us to infer the general strength of the evidence.

          Four outcomes are possible in an Eco-Evidence analysis: “Support for hypothesis”, “Support for alternative hypothesis”, “Inconsistent evidence”, or “Insufficient evidence”. These outcomes are logically based in Popperian ‘falsification’ model of scientific progress [25]. In order to achieve “Support for hypothesis”, at least 20 evidence points must lie in favour of the hypothesis, with fewer than 20 points opposing the hypothesis. This outcome means that the hypothesis is retained as a working hypothesis, but may be falsified in future by further evidence (i.e.: new research). “Support for alternative hypothesis” is a falsification of the cause-effect hypothesis, resulting when 20 or more evidence points oppose the hypothesis, and fewer than 20 points support the hypothesis. The cause-effect hypothesis is also be falsified when at least 20 points both support and oppose the hypothesis, and a result of “Inconsistent evidence” is returned (see below). In these cases, the hypotheses must be disregarded or refined, and a new hypothesis formulated. The conclusion of “Insufficient evidence” is achieved when fewer than 20 points support and oppose the hypothesis, and no further relevant studies can be found, implying a knowledge gap in the literature. With our protocol, we will not reach a conclusion of “Insufficient evidence” unless all search hits have been evaluated, or unless (if there are many thousands of hits from the literature search) the rate of relevant studies being found makes it vanishingly unlikely that we could reach the 20-point threshold. It is very unlikely that we would reach a preliminary conclusion of “Insufficient Evidence” with 20 studies, because of the total evidence weight this implies.

          The logic behind the use of thresholds can be demonstrated with a statistical analogy. With a larger sample size, a hypothesis can be more reliably rejected even if the data are identically distributed. For example, a 2-sample t-value of 2.0 is significantly different from an effect size of 0 with a sample size of 10 (p=0.04, one tail test), but the null hypothesis is retained with a sample size of 5 (p=0.07). Thresholds can therefore be more informative than a simple ratio of evidence, but only when considered in conjunction with the ratio. The Eco Evidence outcomes, like p-values for significance testing, should not be applied without consideration. For example, if 20 evidence points support a hypothesis, and 19 evidence points oppose it, a judgement of “Support for hypothesis” is unreasonable, and the evidence should be judged as inconsistent.

          A finding of “Inconsistent Evidence” may call for a re-examination the initial conceptual model. Often, the reason for an inconsistency is revealed during the evidence review; a flow component may only be associated with an ecological response under certain environmental conditions. For example, if we find “Inconsistent Evidence” for a linkage that combined evidence for woody and herbaceous vegetation (e.g. Do seasonal overbank flows increase germination rates in native riparian vegetation?), we may refine our conceptual model based on vegetation types (woody or herbaceous). When conceptual models are redefined, the literature review (data extraction) is continued. However, if the question is already at the appropriate level of conceptual resolution, the finding of “Inconsistent Evidence” is a final result; the interpretation is that the original hypothesis is falsified, or that the cause-effect linkage only applies under certain circumstances.

          In the last step of the data synthesis (Step 8), we will collectively consider the conclusions for each cause-effect linkage in order to answer the primary question. An overall finding of support for the primary question does not necessarily require support for each of the cause-effect hypotheses considered (e.g. [30]). We will be able to determine whether there is sufficient evidence that the delivery of overbank flows and more high flows will increase the abundance of native riparian vegetation on banks. More importantly, we will be able to identify certain environmental conditions or taxonomic groups that may respond differently. In the discussion of the results, we will provide a narrative synthesis of the studies included in the review, with a particular focus on these two topics. This systematic approach will better inform environmental flows science by determining which flow components should be targeted by environmental flows in order to improve native riparian vegetation.

          Finally, we will assess which studies included in the Eco Evidence analysis could have also been included in a quantitative meta-analysis (i.e. which studies provide the necessary summary statistics). We will test whether the subset of studies suitable for meta-analysis produce different Eco Evidence outcomes than the total set of studies reviewed. We will also tabulate this subset of studies to determine whether they i) differ in study quality and ii) are any more or less likely to support each hypothesis, compared to the total set of studies reviewed. This analysis will form the basis for discussion of the benefits of narrative synthesis compared to meta-analysis.


          aEco Evidence is a new systematic review method currently under consideration for inclusion in the CEE Systematic Review guidelines.


          Authors’ Affiliations

          Department of Infrastructure Engineering, The University of Melbourne
          Department of Resource Management and Geography, The University of Melbourne


          1. Gleick PH: Global freshwater resources: Soft-path solutions for the 21st century. Science 2003,302(5650):1524–1528. 10.1126/science.1089967View Article
          2. Konrad CP, Warner A, Higgins JV: Evaluating dam re-operation for freshwater conservation in the Sustainable Rivers project. River Res Appl 2012,28(6):777–792. 10.1002/rra.1524View Article
          3. Merritt DM, Scott ML, Poff NL, Auble GT, Lytle DA: Theory, methods and tools for determining environmental flows for riparian vegetation: riparian vegetation-flow response guilds. Freshwater Biol 2010,55(1):206–225. 10.1111/j.1365-2427.2009.02206.xView Article
          4. Souchon Y, Sabaton C, Deibel R, Reiser D, Kershner J, Gard M, Katopodis C, Leonard P, Poff NL, Miller WJ, Lamb BL: Detecting biological responses to flow management: Missed opportunities; Future directions. River Res Appl 2008,24(5):506–518. 10.1002/rra.1134View Article
          5. Bernhardt ES, Palmer MA, Allan JD, Alexander G, Barnas K, Brooks S, Carr J, Clayton S, Dahm C, Follstad-Shah J, Galat D, Gloss S, Goodwin P, Hart D, Hassett B, Jenkinson R, Katz S, Kondolf GM, Lake PS, Lave R, Meyer JL, O'Donnell TK, Pagano L, Powell B, Sudduth E: Synthesizing US river restoration efforts. Science 2005,308(5722):636–637. 10.1126/science.1109769View Article
          6. DECC: New South Wales RiverBank business plan part A: program plan 2006–2011. Sydney South, Australia: Department of Environment and Climate Change NSW; 2008. Available at http://​tinyurl.​com/​nc358c
          7. DEWHA: Planned approach for restoring the balance in the Murray-Darling Basin. Water purchase program: 2008–09 and beyond. Canberra, Australia: Department of the Environment, Water, Heritage, and the Arts; 2008. Available at http://​tinyurl.​com/​yavrfgg
          8. DSE: Our water our future: the next stage of hte government's water plan. Melbourne, Australia: Department of Sustainablity and the Environment; 2007. Available at http://​tinyurl.​com/​nnm6vm
          9. Bunn SE, Arthington AH: Basic principles and ecological consequences of altered flow regimes for aquatic biodiversity. Environ Manage 2002,30(4):492–507. 10.1007/s00267-002-2737-0View Article
          10. Lloyd N, Quinn G, Thoms M, Arthington A, Gawne B, Humphries P, Walker K: Does Flow Modification Cause Geomorphological and Ecological Response in Rivers? A Literature Review From an Australian Perspective. Technical Report 1 /2004 CRC for Freshwater Ecology. 2003.
          11. Nilsson C, Svedmark M: Basic principles and ecological consequences of changing water regimes: Riparian plant communities. Environ Manage 2002,30(4):468–480. 10.1007/s00267-002-2735-2View Article
          12. Poff NL, Allan JD, Bain MB, Karr JR, Prestegaard KL, Richter BD, Sparks RE, Stromberg JC: The natural flow regime: a paradigm for river conservation and restoration. Bioscience 1997,47(11):769–784. 10.2307/1313099View Article
          13. Poff NL, Zimmerman JKH: Ecological responses to altered flow regimes: a literature review to inform the science and management of environmental flows. Freshwater Biol 2010,55(1):194–205. 10.1111/j.1365-2427.2009.02272.xView Article
          14. VEWH, Authority G-HCM and Authority WCM: Seasonal Watering Plan 2011–12, Schedule 7: Wimmera-Glenelg system. Victoria, Australia: Victorian Environmental Water Holder; 2011.
          15. Stromberg JC: Frémont Cottonwood-Goodding Willow Riparian Forests: A Review of Their Ecology, Threats, and Recovery Potential. J Ariz Nev Acad Sci 1993,27(1):97–110.
          16. BLM: Record of Decision and Green River Resource Management Plan. USA: US Department of the Interior, Bureau of Land Management; 1997.
          17. Lytle DA, Poff NL: Adaptation to natural flow regimes. Trends Ecol Evol 2004,19(2):94–100. 10.1016/j.tree.2003.10.002View Article
          18. Stromberg JC, Lite SJ, Marler R, Paradzick C, Shafroth PB, Shorrock D, White JM, White MS: Altered stream-flow regimes and invasive plant species: the Tamarix case. Global Ecol Biogeogr 2007,16(3):381–393. 10.1111/j.1466-8238.2007.00297.xView Article
          19. Chee YE, Webb JA, Stewardson M, Cottingham P: Victorian Environmental Flows Monitoring and Assessment Program: Monitoring and assessing environmental flow releases in the Thompson River. Canberra, Australia: eWater Cooperative Research Centre; 2009. Available at http://​tinyurl.​com/​VEFMAP-Thomson
          20. Cottingham P, Stewardson M, Webb A: Victorian environmental flows monitoring and assessment program. Stage 1: statewide framework. Australia: CRC for Freshwater Ecology and CRC for Catchment Hydrology; 2005. Available at http://​tinyurl.​com/​2feys3
          21. Webb JA, Stewardson MJ, Chee YE, Schreiber ESG, Sharpe AK, Jensz MC: Negotiating the turbulent boundary: the challenges of building a science-management collaboration for landscape-scale monitoring of environmental flows. Mar Freshwater Res 2010,61(7):798–807. 10.1071/MF09059View Article
          22. Tugwell P, Haynes RB: Assessing claims of causation. In Clinical epidemiology: how to do clinical practice research. Edited by: Haynes RB, Sackett DL, Guyatt GH, Tugwell P. Philadelphia, Pennsylvania: Lippincott, Williams and Wilkins; 2006:356–387.
          23. Weed DL: On the use of causal criteria. Int J Epidemiol 1997,26(6):1137–1141. 10.1093/ije/26.6.1137View Article
          24. Centre for Evidence-Based Conservation: Guidelines for Systematic Review in Environmental Management. Version 4.0. Environmental Evidence; 2010. http://​www.​environmentalevi​dence.​org/​Authors.​htm
          25. Norris RH, Webb JA, Nichols SJ, Stewardson MJ, Harrison ET: Analyzing cause and effect in environmental assessments: using weighted evidence from the literature. Freshwater Sci 2012,31(1):5–21. 10.1899/11-027.1View Article
          26. Webb JA, Wealands SR, Lea P, Nichols SJ, de Little SC, Stewardson MJ, Norris RH: Eco Evidence: using the scientific literature to inform evidence-based decision making in environmental management. In MODSIM2011 International Congress on Modelling and Simulation: 2011; Perth, Australia. Edited by: Chan F, Marinova D, Anderssen RS. Modelling and Simulation Society of Australia and New Zealand; 2011.
          27. Nichols S, Webb A, Norris R, Stewardson M: Eco Evidence analysis methods manual: a systematic approach to evaluate causality in environmental science. Canberra, Australia: eWater Cooperative Research Centre; 2011. Available at http://​tinyurl.​com/​Eco-Evidence-manual
          28. Norris R, Nichols S, Ransom G, Webb A, Stewardson M, Liston P, Mugodo J: Causal criteria analysis methods manual: a systematic approach to evaluate causality in environmental science. Canberra, Australia: eWater Cooperative Research Centre; 2008. Available at http://​tinyurl.​com/​CCManual
          29. Roberts PD, Stewart GB, Pullin AS: Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol Conserv 2006,132(4):409–423. 10.1016/j.biocon.2006.04.034View Article
          30. Greet J, Webb JA, Cousens RD: The importance of seasonal flow timing for riparian vegetation dynamics: a systematic review using causal criteria analysis. Freshwater Biol 2011,56(7):1231–1247. 10.1111/j.1365-2427.2011.02564.xView Article
          31. Bekkering GE, Harris RJ, Thomas S, Mayer AMB, Beynon R, Ness AR, Harbord RM, Bain C, Smith GD, Sterne JAC: How much of the data published in observational studies of the association between diet and prostate or bladder cancer is usable for meta-analysis? Am J Epidemiol 2008,167(9):1017–1026. 10.1093/aje/kwn005View Article
          32. Grove JR, Webb JA, Marren PM, Stewardson MJ, Wealands SR: High and dry: comparing literature review approaches to reveal the data that informs the geomorphic management of regulated river floodplains. Wetlands 2012, 32: 215–224. 10.1007/s13157-011-0253-9View Article
          33. Harrison ET: Fine sediment in rivers: scale of ecological outcomes. Canberra, Australia: University of Canberra; 2010. Ph.D. thesis
          34. Miller KA, Webb JA, de Little SC, Stewardson MJ: Environmental flows can reduce the encroachment of terrestral vegetation into river channels: a systematic literature review. Environ Manage in review .
          35. Webb JA, Wallis EM, Stewardson MJ: A systematic review of published evidence linking wetland plants to water regime components. Aquat Bot 2012. 10.1016/j.aquabot.2012.1006.1003
          36. Landis JR, Koch GG: Measurement of Observer Agreement for Categorical Data. Biometrics 1977,33(1):159–174. 10.2307/2529310View Article
          37. U.S. Department of Health and Human Services: The Health Consequences of Smoking: A Report of the Surgeon General. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health; 2004.

          This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.