Skip to main content

Potential types of bias when estimating causal effects in environmental research and how to interpret them

Abstract

To inform environmental policy and practice, researchers estimate effects of interventions/exposures by conducting primary research (e.g., impact evaluations) or secondary research (e.g., evidence reviews). If these estimates are derived from poorly conducted/reported research, then they could misinform policy and practice by providing biased estimates. Many types of bias have been described, especially in health and medical sciences. We aimed to map all types of bias from the literature that are relevant to estimating causal effects in the environmental sector. All the types of bias were initially identified by using the Catalogue of Bias (catalogofbias.org) and reviewing key publications (n = 11) that previously collated and described biases. We identified 121 (out of 206) types of bias that were relevant to estimating causal effects in the environmental sector. We provide a general interpretation of every relevant type of bias covered by seven risk-of-bias domains for primary research: risk of confounding biases; risk of post-intervention/exposure selection biases; risk of misclassified/mismeasured comparison biases; risk of performance biases; risk of detection biases; risk of outcome reporting biases; risk of outcome assessment biases, and four domains for secondary research: risk of searching biases; risk of screening biases; risk of study appraisal and data coding/extraction biases; risk of data synthesis biases. Our collation should help scientists and decision makers in the environmental sector be better aware of the nature of bias in estimation of causal effects. Future research is needed to formalise the definitions of the collated types of bias such as through decomposition using mathematical formulae.

Background

Estimation of average causal effects of interventions or exposures can provide important evidence to inform environmental policy and practice. There is a risk that if these estimates are derived from poorly conducted (or selective reporting or selectively reported [1,2,3,4]) primary (e.g., impact evaluation studies) or secondary research (e.g., evidence reviews), then they will potentially misinform policy and practice. Biased estimates of average causal effects (inaccuracy) are more problematic than imprecise estimates of average causal effects due to random error which may be measured by standard errors or confidence intervals [5] because a bias is a systematic error or ‘deviation from the truth’ [6, 7], and can be a direct threat to the validity of a causal inference [8]. Biases can be quantified only when the true average causal effects are known, or when hypothetical true (or expected) average effects are set as reference values (i.e., B = mµ, where m is an estimated average causal effect and µ is the true causal effect [7, 9, 10]; it is also defined as \(E\left( {\widehat{\theta }} \right) - \theta\) [8]). This means that, when there is a systematic error, an estimated average causal effect is inaccurate and either an under- or overestimate of the true average causal effect (i.e., m = µ ± B, where B ≠ 0 [10]). However, distinguishing biases from random errors and quantification of biases may not be possible using real-world data [6]. Instead of quantifying biases, the extent of ‘risk of bias’ or ‘threats to internal validity’ may need to be communicated explicitly [11,12,13].

Many types of bias (specific phenomena resulting in systematic errors described in the literature and on the Internet) have been described, most notably in medical and health sciences where methods for assessing the risk of bias originated (e.g., Catalogue of Bias describes 62 types of bias as of 29 August 2021: catalogofbias.org). For example, observer bias is described as ‘the process of observing and recording information which includes systematic discrepancies from the truth’ [14], and detection bias is described as ‘systematic differences between groups in how outcomes are determined’ [15]. Types of bias, which are considered by individuals or groups as distinctive phenomena that can give rise to biased estimates, are generally described freely, but some can also be defined formally using mathematical languages [e.g., confounding bias and collider bias are often defined using directed acyclic graphs (DAGs)] [16, 17].

In the health sector, there are dedicated tools for assessing the risk of bias or internal validity [18, 19]. For example, Risk of Bias version 2 (RoB 2) [20] and Risk Of Bias In Non-randomised Studies—of Interventions (ROBINS-I) [21] are widely recommended domain-based tools for assessing the risk of bias in primary research (the former is for randomised controlled trials and the latter is for non-randomised intervention studies) when conducting systematic reviews in the health sector [18]. There is another widely recommended domain-based tool called Risk Of Bias In Systematic reviews (ROBIS) which is for specifically assessing the risk of bias arising from the conduct of secondary research in the health sector [18, 22]. These risk-of-bias assessment tools guide users to focus on a few ‘domains’ of bias (broad categories of bias specifically designed for assessing the risk of bias in particular situations; for example, when you plan to synthesise effect sizes from multiple studies, you would need to assess the risk of bias in primary research assessing effects by developing a comprehensive set of domains to address all potential sources of bias occurring in primary research in your subject area) [13]. Risk-of-bias assessors can make overall judgements about the extent of risk of bias for the specified domains (e.g., low risk of bias, some concerns, or high risk of bias [20]). The concept of domains of bias was initially developed in the health sector to assess the risk of bias for groups of types of bias that can occur at specific stages of primary or secondary research [20, 21]. For example, in the ROBINS-I tool, detection bias, recall bias, information bias, misclassification bias, observer bias, and measurement bias are all included in a domain called ‘risk of bias in measurement of outcome’ [21].

Following the developments of the risk-of-bias assessment tools in the health sector, the Collaboration for Environmental Evidence (hereafter ‘CEE’: environmentalevidence.org) is adopting such a domain-based approach and has been developing a domain-based critical appraisal tool called ‘CEE Critical Appraisal Tool’ since 2020. This tool is for assessing the risk of bias in primary research and is available for use in evidence reviews in the environmental sector (environmentalevidence.org/cee-critical-appraisal-tool). Employing domains of bias is now strongly encouraged when conducting evidence reviews in medical and health sciences [20, 21, 23, 24] as well as in the environmental sector [13] because assessing every individual type of bias is laborious and there are no universal definitions of all existing types of bias in estimation of causal effects in science. A recent review on critical appraisal in ecology highlighted this important development and the urgent need to evaluate the validity and reliability of the CEE Critical Appraisal Tool because, remarkably, only 4% of reviews labelled as ‘systematic reviews’ in ecology actually conducted critical appraisal [25]. This finding suggests insufficient education about biases in environmental science courses and a lack of understanding of how to avoid providing potentially biased estimates [26].

We aim to raise awareness about biases in the environmental sector by mapping types of bias that may impact the reliability of the results of environmental research evaluating the effects of interventions or exposures and categorising them into the developed domains of bias. The objectives of our research are: (1) to identify potential types of bias, that have been previously collated, in estimation of causal effects in either primary or secondary research; (2) to evaluate the relevance of the identified biases to the environmental sector, which incorporates not only non-human biotic (e.g., plants, animals) and abiotic subjects (e.g., water, atmosphere) but also human subjects; and (3) to utilise the resultant list of biases to evaluate the coverage of the domains of bias for a previously developed assessment tool to use in environmental research, namely the CEE Critical Appraisal Tool.

To avoid confusion, when we refer to domains of bias, we use plural rather than singular (e.g., confounding biases) as noted below. Also, note that we do not aim to evaluate relatedness among biases (e.g., we do not discuss whether selection bias is related to attrition bias [27]). Rather we aim to provide an overview of the types of bias from the view of managing the risk of bias by employing the developed domains.

Methods

Listing the types of bias previously collated and described elsewhere

In order to develop a comprehensive list of potential biases we used the Catalogue of Bias (catalogofbias.org), and reviewed 11 key publications in the field: Sackett [28], Bayliss and Beyer [29], Clarke et al. [30], Smith and Noble [31], Thakur et al. [32], Paradis [33], Warden [34], Delegado-Rodriguez and Llorca [35], Hartman et al. [36], Marchevsky [37], and Pannucci and Wilkins [38]. The Catalogue of Bias is an ongoing collaborative project aiming to map all the biases that affect health and medical evidence. The Catalogue of Bias platform was developed by the Centre for Evidence-Based Medicine (CEBM) at Oxford University through reviewing the literature and regular meetings. We considered the Catalogue of Bias to provide a robust and comprehensive source. To our knowledge, there was no equivalent database of bias in environmental sciences, so the Catalogue of Bias was the only choice, apart from the 11 key publications. The additional 11 key publications used as information sources were selected because they were known to the authors as seminal papers that collated and described biases. We listed biases that were described in these sources in a Microsoft Excel spreadsheet (Redmond, WA, USA). The intention here was to identify the breadth of biases that have been recognised or described and not to identify all studies describing each bias. We therefore did not conduct a systematic literature search. The broad scope of our review and the time and resources available also precluded this.

Evaluating the relevance of the biases to estimation of effects in the environmental sector

Since the vast majority of the types of bias were described in the medical and health research contexts, we evaluated them for relevance to estimation of effects of interventions or exposures in the environmental sector. Relevance was evaluated by a single reviewer (KK) using the algorithm provided in Fig. 1.

Fig. 1
figure 1

The algorithm used for evaluating the relevance of biases described in the sources

We employed a broad and all-inclusive scope for evaluating relevance to the environmental sector. The following a priori list of relevant topics was developed and used for assessing relevance: agriculture, aquaculture, biodiversity conservation, climate change, ecology and evolution, ecosystem services, environment and human wellbeing, environmental education, environmental legislation, fisheries, food security, forestry, invasive species, natural resource management, park and protected area management, pollution, soil management, sustainable energy use, waste management, wastewater management, water security, as well as any research relevant to environmental sustainability. The broad scope is consistent with that of CEE. The scope inherently included not only non-human biotic and abiotic subjects but also human subjects, however, human health outcomes were out of scope. The algorithm was applied by one person (KK) and the resulting decisions were checked by a second person (AP), and any disagreements or uncertainties were discussed among all authors. We highlight that we did not carry out any consistency checking, as we did not aim to aggregate effects, and did not aim to provide databases for aggregating effects. Hence, Cohen’s kappa statistic [39, 40] was not calculated.

Evaluating the coverage of the developed domains of bias

Suzuki et al. [8] provided an organisational schema for systematic error in causal inference in primary research. In primary research, systematic error can be divided into structural error and analytic error. Structural error can be further divided into bias relating to measurement of intervention, exposure or outcome, and bias relating to exchangeability. Exchangeability refers to independence between the counterfactual outcome \({Y}^{a}\) (the outcome which a subject or area would have experienced if it had received a specified intervention or exposure value [41]) and the observed intervention or exposure \(A\), and it is denoted as Ya A, for all treatment values a (all values of the treatment variable that may differ among individual subjects or areas) [42]. Alternatively, exchangeability refers to an assumption that the outcome in the comparator group would mirror the outcome in the intervention or exposure group if the subjects or areas in the comparator group had been subjected to the same intervention or exposure as those in the intervention or exposure group [17]. Analytic error is a systematic error that cannot be explained by structural error, and it includes bias relating to reporting [20, 21] and statistical methods [23, 24, 43]. Figure 2 illustrates our assumptions about the organisation of bias occurring in primary and secondary research from the broadest term (systematic error) to the developed domains of bias.

Fig. 2
figure 2

Organisation of bias from the broadest term (systematic error) to the developed domains of bias

Frampton et al. [13] provided the principles of risk-of-bias assessment in the environmental sector and recommended seven domains (or classes) to be assessed when conducting evidence reviews assessing the effect of an intervention or exposure. These domains for primary research were consistent with the CEE Critical Appraisal Tool version 0.3 (see environmentalevidence.org/cee-critical-appraisal-tool for more details) as including the following:

  1. 1.

    Confounding biases: Risk of biases due to an uncontrolled (or inappropriately controlled) variable (confounder) that influences both the intervention/exposure and the outcome

  2. 2.

    Post-intervention/exposure selection biases: Risk of biases arising from systematic differences in the selection of subjects or areas into the study or analysis after the intervention or exposure

  3. 3.

    Misclassified/mismeasured comparison biases: Risk of biases arising from misclassification or mismeasurement of the intervention, exposure and/or comparator (applicable to observational studies only)

  4. 4.

    Performance biases: Risk of biases due to altered treatment procedure of interest (applicable to experimental studies only; treatment procedures are series of actions for applying the experimental (intervention or exposure) treatment [44]; note that an experimental treatment can be either an intervention or an exposure, or both, depending on the outcome measure. For example, when a fertilizer is applied to a crop, this is an intervention if the outcome measure is crop yield, but an exposure if the outcome measure is change in soil microfauna [45])

  5. 5.

    Detection biases: Risk of biases arising from systematic differences in measurement of outcomes

  6. 6.

    Outcome reporting biases: Risk of biases in reporting of study findings

  7. 7.

    Outcome assessment biases: Risk of biases due to error in applied statistical methods

Regarding the domains for secondary research, we used the CEE Synthesis Appraisal Tool (CEESAT) version 2 [46, 47]. CEESAT is an eight-criteria checklist consisting of sixteen questions for assessing environmental evidence reviews in terms of risk of bias, repeatability, and transparency, and hence not all the criteria are relevant to risk of bias (e.g., review question setting and provision of limitations). Using only the relevant criteria resulted in four domains of bias which were the same as the ROBIS tool [22], and thus consistent with a widely recommended tool in medical and health sciences [18]:

  1. 1.

    Searching biases: Risk of biases in searches of relevant records

  2. 2.

    Screening biases: Risk of biases arising from screening of potentially relevant records

  3. 3.

    Study appraisal and data coding/extraction biases: Risk of biases due to the lack of or inappropriate conduct of study appraisal and data coding/extraction

  4. 4.

    Data synthesis biases: Risk of biases due to employing inappropriate synthesis methods

Each type of bias was checked by a single reviewer (KK) for relevancy against each domain. Each bias-and-domain combination was data-coded in the database (spreadsheet), by the single reviewer using subjective judgements (‘Yes’ = relevant/‘No’ = not relevant). All decisions were double-checked by the other authors. The evaluator was allowed to take special notes when some types of bias were beyond the provided domains but there were no such cases. We also data-coded relevant levels of research (primary and/or secondary research) for all included types of bias.

Results

Types of bias relevant to the environmental sector

Figure 3 summarises the results of the bias selection process and provides the numbers of biases identified in each of the sources searched. Some biases were described by multiple sources while some biases were described by one source only. There were 206 types of bias after merging 53 bias descriptors that were specifically described as synonyms; we did not merge bias descriptors unless they were specifically referred to as synonyms to make sure none are missed from the list. Note that we did not reproduce (verbatim) the descriptions of the biases provided in the sources mainly due to copyright restrictions.

Fig. 3
figure 3

Identification of biases previously collated and described elsewhere

After the evaluation of relevance, we identified 121 types of bias (accompanied by 24 synonyms specified in the sources) that were relevant to estimation of effects in the environmental sector. Of the 85 excluded types of biases, 53 (62%) were excluded due to a lack of relevance to estimation of effects. Thirty-one (37%) were excluded due to a lack of relevance to the environmental sector; the most common reasons for exclusions were specifically referring to human health outcomes, diagnosis, and healthcare and medical interventions. One (1%; culture bias described by Warden 2015 [34]) was excluded because its description was insufficient for judging its relevance. All the excluded types of bias can be found in Additional file 1.

The general interpretations and relevant domains of the 121 types of bias relevant to the environmental sector are provided in Table 1. Sixty-eight types of bias (56%) were relevant to primary research only and 18 types of bias (15%) were relevant to secondary research only. Thirty-five types of bias (29%) were relevant to both the primary and secondary research. We provide the database of these biases in Additional file 2 so that readers can use the filter function of the spreadsheet to find types of bias relevant to specific domains. To increase its convenience, we also created a sheet to list types of bias relevant to each domain.

Table 1 Types of bias relevant to the environmental sector in the alphabetical order

The coverage of the domains of bias

There was no type of bias that was not applicable to any domain, and we did not find any types of bias that were beyond the domains, suggesting that domains were sufficient to cover the 121 types of bias. Eighty-six types of bias (71%) were applicable to multiple domains while 35 types of bias (29%) were applicable to only one domain of bias.

Discussion

Overview of findings

We found 121 (out of 206) types of bias that were relevant to estimation of causal effects in the environmental sector. We provided a general interpretation of every relevant type of bias covered by seven risk-of-bias domains for primary research and four domains for secondary research.

Strengths and limitations

As far as we are aware this is the first study to provide a comprehensive map of the type of bias that may affect the reliability of the results of environmental research studies and reviews. It provides an important summary of the breadth of biases which may impact the findings. There are limitations that should be noted. We used a pragmatic approach to identify potentially relevant biases by utilising the Catalogue of Bias and 11 key publications. We did not conduct a systematic literature search, similar to the approach used by some of the key documents we consulted [28,29,30,31,32,33, 35,36,37,38]. However, the Catalogue of Bias which is developed by CEBM at Oxford University and is continually updated, is considered a robust source for identifying biases that affect health and medical evidence. There may be other relevant types of bias that are collated and described elsewhere, but we are not aware of any published studies specific to the environmental sector that aimed to identify types of bias. We used subjective judgements for evaluating relevance and some aspects of biases were open for interpretation although we conducted independent double checking. We provided only ‘general’ interpretations of the types of bias in this paper.

Implications for research and practice

Since education about biases is not sufficient and the level of knowledge about biases are concerning in the environmental sector [26], we hope the provision of the general interpretations of the 121 types of bias helps environmental scientists, as well as decision-makers, be better aware of the biases. When environmental scientists, practitioners, and policymakers come across certain types of bias when they read research or review papers, or when they communicate about certain types of bias, we suggest they check our dedicated list of biases relevant to estimation of causal effects in the environmental sector.

Although some types of bias such as confounding bias, selection bias, and measurement bias have been formally defined [8, 17], future research is needed to formalise definitions of other types of bias such as through decomposition using mathematical formulae and/or by employing directed acyclic graphs (DAGs). For example, Suzuki et al. 2016 [8] decomposed confounding bias by comparing the causal measure (A → Y), µ(E(Y1), E(Y0)), and the situation where a confounding variable is present as:

$$\mu \left( {E\left( {Y{|}A = 1} \right), E\left( {Y{|}A = 0} \right)} \right) - \mu \left( {E\left( {Y_{1} } \right),E\left( {Y_{0} } \right)} \right)$$

where Y refers to outcome and A refers to intervention or exposure (A = 1 receives the intervention or exposure and A = 0 does not receive the intervention or exposure). Although the practicality of defining all biases remains to be evaluated, formalising definitions is particularly important because, to some extent, vagueness remains in the descriptions of the biases. This would also help identify duplication and cross over between biases identified by different researchers.

As the types of bias are described theoretically, future research is also needed for collating empirical studies that evaluate the impacts of the identified biases. Such research is desired even in the health sector for better managing the risk of bias [86]. In the environmental sector, for example, Hudson et al. [87] and Takeshita et al. [88] evaluated the potential extent of selection bias and confounding bias, respectively. Konno et al. [89] and Konno and Pullin [90] evaluated the potential extent of language bias and availability bias, respectively. If such studies are collated and summarised, environmental scientists should be better able to manage the risk of bias. We hope this paper guides future research.

A lack of and inappropriate conduct of risk-of-bias assessment in the environmental sector has been reported in the literature [13, 25, 91]. Our findings showed that the domains selected for use in CEE Critical Appraisal Tool and CEE Synthesis Assessment Tool are sufficient and cover all the biases identified in primary and secondary research (Fig. 2) and can be used to facilitate communication about the risk of bias. A domain-based tool should incorporate a series of checklist questions within each domain that prompts users to consider relevant types and sources of bias. Developing a series of checklist questions requires relevant expertise [20, 21], and thus, we encourage evidence providers (e.g., impact evaluators, evidence synthesists) as well as evidence consumers (e.g., decision-makers) to make use of the existing tools for assessing the risk of bias (e.g., the CEE Critical Appraisal Tool for primary research and the CEE Synthesis Appraisal Tool for secondary research), rather than developing a tool on their own, as it may result in inappropriate conduct of risk-of-bias assessment [13, 91]. However, at the time of writing, the CEE Critical Appraisal Tool is still under development, and we hope users of this tool will provide their feedback for the developers. The most important aspect for further development of the tool is inter-rater reliability (i.e., consistency between assessors) which can be quantified using a kappa statistic [92] and reported in eventual publications (e.g., Systematic Reviews [93], Systematic Maps [94]). Even in medical and health sciences, inter-rater reliability is still an issue [95, 96].

As part of our future work, we plan to consult researchers and policymakers to reach a consensus on the list of biases for better and active communication within the communities. We highlight the importance of including appropriately qualified and experienced stakeholders in any consultation because it is likely that many people working in environmental research and policy are inadequately familiar with biases [26], and thus there is a need to educate environmental scientists and decision-makers about the types of bias. There is also a need to provide training and develop capacity for use of risk-of-bias assessment tools in the environmental sciences. We are not aware of any institution that teaches this subject in environmental contexts, even at Masters level. In response to this gap, CEE intends to develop a series of online training videos on critical appraisal. If many more scholars, institutions as well as learned societies can help the environmental sector better understand the nature of biases, more active communication of the risk of bias may be achieved.

Availability of data and materials

The datasets generated during the current study are available in Additional files.

References

  1. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ (Online). 2010;340: c365.

    PubMed  Google Scholar 

  2. Fleming PS, Koletsi D, Dwan K, Pandis N. Outcome discrepancies and selective reporting: Impacting the leading journals? PLoS ONE. 2015;10: e0127495.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: Reanalysis of meta-analyses. BMJ. 2012;344: d7202.

    Article  PubMed  Google Scholar 

  4. Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:1–12.

    Article  CAS  Google Scholar 

  5. Fox MP, MacLehose RF, Lash TL. Applying quantitative bias analysis to epidemiologic data. 2nd ed. Cham: Springer; 2021.

    Book  Google Scholar 

  6. Eisenhart C. Expression of the uncertainties of final results. Science. 1979;1968(160):1201–4.

    Google Scholar 

  7. Everitt BS, Skrondal A. The Cambridge dictionary of statistics. The Cambridge Dictionary of Statistics; 2010.

    Book  Google Scholar 

  8. Suzuki E, Tsuda T, Mitsuhashi T, Mansournia MA, Yamamoto E. Errors in causal inference: an organizational schema for systematic error and random error. Ann Epidemiol. 2016;26:788-793.e1.

    Article  PubMed  Google Scholar 

  9. Cochran W. Sampling techniques. 3rd ed. New York: John Wiley and Sons; 1977.

    Google Scholar 

  10. Upton G, Cook I. A dictionary of statistics. 3rd ed. New York: Oxford University Press; 2014.

    Google Scholar 

  11. Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane; 2019. www.training.cochrane.org/handbook

  12. Collaboration for Environmental Evidence. Guidelines and Standards for Evidence Synthesis in Environmental Management Version 5.0. 2018;

  13. Frampton G, Whaley P, Bennett M, Bilotta G, Dorne JLCM, Eales J, et al. Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews. Environ Evid. 2022;11:12.

    Article  Google Scholar 

  14. Mahtani K, Spencer E, Brassey J. Observer bias. Catalogue of bias; 2017. https://catalogofbias.org/biases/observer-bias/. Accessed 5 Oct 2021.

  15. O’Sullivan J, Banerjee A, Pluddemann A. Detection bias. Catalogue of Bias. 2017 [cited 2021 Oct 5]. https://catalogofbias.org/biases/detection-bias/. Accessed 5 Oct 2021.

  16. Pearl J. Causality. 2nd ed. New York: Cambridge University Press; 2009.

    Book  Google Scholar 

  17. Hernán MA, Robins JM. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC; 2020. https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/

  18. Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res. 2020;7:1–11.

    Google Scholar 

  19. Pullin AS, Knight TM. Effectiveness in conservation practice: pointers from medicine and public health. Conserv Biol. 2001;15:50–4.

    Article  Google Scholar 

  20. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. The BMJ. 2019;366: l4898.

    Article  PubMed  Google Scholar 

  21. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Online). 2016;355: i4919.

    PubMed  Google Scholar 

  22. Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Barker TH, Stone JC, Sears K, Klugar M, Tufanaru C, Leonardi-Bee J, et al. The revised JBI critical appraisal tool for the assessment of risk of bias for randomized controlled trials. JBI Evid Synth. 2023;21:494–506.

    Article  PubMed  Google Scholar 

  24. Barker TH, Stone JC, Sears K, Klugar M, Leonardi-Bee J, Tufanaru C, et al. Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process. JBI Evid Synth. 2023;21:478–93.

    Article  PubMed  Google Scholar 

  25. Stanhope J, Weinstein P. Critical appraisal in ecology: What tools are available, and what is being used in systematic reviews? Res Synth Methods. 2022;14:342–56.

    Article  PubMed  Google Scholar 

  26. Zvereva EL, Kozlov MV. Biases in ecological research: attitudes of scientists and ways of control. Sci Rep. 2021;11:342–56.

    Article  Google Scholar 

  27. Nunan D, Aronson J, Bankhead C. Catalogue of bias: attrition bias. BMJ Evid Based Med. 2018;23:21–2.

    Article  PubMed  Google Scholar 

  28. Sackett DL. Bias in analytic research. J Chronic Dis. 1979;32:51–63.

    Article  CAS  PubMed  Google Scholar 

  29. Bayliss HR, Beyer FR. Information retrieval for ecological syntheses. Res Synth Methods. 2015;6:136–48.

    Article  PubMed  Google Scholar 

  30. Clarke M, Atkinson P, Badenoch D, Chalmers I, Glasziou P, Podolsky S, et al. The James Lind library’s introduction to fair tests of treatments. www.jameslindlibrary.org

  31. Smith J, Noble H. Bias in research. Evidence Based. Nursing. 2014;17:100–1.

    Google Scholar 

  32. Thakur A, Choudhury V, Saluja S. Bias in research. Curr Med Res Pract. 2012;2:106–10.

    Google Scholar 

  33. Paradis C. Bias in surgical research. Ann Surg. 2008;248:180–8.

    Article  PubMed  Google Scholar 

  34. Warden G. Definitions of bias in clinical research. Clin Epidemiol. 2015;2249:31–48.

    Article  Google Scholar 

  35. Delgado-Rodriguez M, Llorca J. Bias. J Epidemiol Community Health (1978). 2004;58:635–41.

    Article  Google Scholar 

  36. Hartman JM, Forsen JW, Wallace MS, Neely JG. Tutorials in clinical research: Part IV: recognizing and controlling bias. Laryngoscope. 2002;112:23–31.

    Article  PubMed  Google Scholar 

  37. Marchevsky D. Bias. Critical appraisal of medical literature. Springer Science & Business Media; 2000. p. 57–61.

    Book  Google Scholar 

  38. Pannucci CJ, Wilkins EG. Identifying and avoiding bias in research. Plast Reconstr Surg. 2010;126:619.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37–46.

    Article  Google Scholar 

  40. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22:276–82.

    Article  PubMed  Google Scholar 

  41. Porta M. A dictionary of epidemiology. 6th ed. New York: Oxford University Press; 2014.

    Book  Google Scholar 

  42. Hernán MA. A definition of causal effect for epidemiological research. J Epidemiol Community Health. 1978;2004(58):265–71.

    Google Scholar 

  43. Steenland K, Schubauer-Berigan MK, Vermeulen R, Lunn RM, Straif K, Zahm S, et al. Risk of bias assessments and evidence syntheses for observational epidemiologic studies of environmental and occupational exposures: Strengths and limitations. Environ Health Perspect. 2020;128: 095002.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Cochran W, Cox G. Experimental designs. 2nd ed. New York: John Wiley & Sons; 1957.

    Google Scholar 

  45. Collaboration for Environmental Evidence. Guidelines and Standards for Evidence Synthesis in Environmental Management Version 5.1. 2022. 2022 . https://environmentalevidence.org/information-for-authors/guidelines-for-authors/. Accessed 3 Mar 2023.

  46. Konno K, Cheng SH, Eales J, Frampton G, Kohl C, Livoreil B, et al. The CEEDER database of evidence reviews: an open-access evidence service for researchers and decision-makers. Environ Sci Policy. 2020;114:256–62.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Woodcock P, Pullin AS, Kaiser MJ. Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: a methodology. Biol Conserv. 2014;176:54–62.

    Article  Google Scholar 

  48. Spencer E, Heneghan C. All’s well literature bias. Catalogue of Bias. 2018. https://catalogofbias.org/biases/alls-well-literature-bias/. Accessed 5 Oct 2021.

  49. Spencer E, Heneghan C, Nunan D. Allocation bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/allocation-bias/. Accessed 3 Apr 2022.

  50. Spencer E, Brassey J. Ascertainment bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/ascertainment-bias/. Accessed 5 Oct 2021.

  51. Spencer E, Mahtani K. Hawthorne effect. Catalogue of Bias. 2017. https://catalogofbias.org/biases/hawthorne-effect/. Accessed 5 Oct 2021.

  52. Bankhead C, Aronson J, Nunan D. Attrition bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/attrition-bias. Accessed 5 Oct 2021.

  53. Banerjee A, Nunan D. Availability bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/availability-bias/. Accessed 5 Oct 2021.

  54. Spencer E, Heneghan C. Chronological bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/chronological-bias/. Accessed 5 Oct 2021.

  55. Lee H, Aronson J, Nunan D. Collider bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/collider-bias/. Accessed 5 Oct 2021.

  56. Spencer E, Heneghan C. Compliance bias. Catalogue of Bias. 2018. https://catalogofbias.org/biases/compliance-bias/. Accessed 5 Oct 2021.

  57. Spencer E, Heneghan C. Confirmation bias. Catalogue of Bias. 2018. https://catalogofbias.org/biases/confirmation-bias/. Accessed 5 Oct 2021.

  58. Aronson J, Bankhead C, Nunan D. Confounding. Catalogue of Bias. 2018. https://catalogofbias.org/biases/confounding/. Accessed 5 Oct 2021.

  59. Aronson J, Bankhead C, Mahtani K, Nunan D. Confounding by indication. Catalogue of Bias. 2018. https://catalogofbias.org/biases/confounding-by-indication/. Accessed 5 Oct 2021.

  60. Erasmus A, Holman B, Ioannidis J. Data-dredging bias. Catalogue of Bias. 2020. https://catalogofbias.org/biases/data-dredging-bias/. Accessed 5 Oct 2021.

  61. Aronson J, Bankhead C, Nunan D. Hot stuff bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/hot-stuff-bias/. Accessed 5 Oct 2021.

  62. Holman B, Bero L, Mintzes B. Industry sponsorship bias. Catalogue of Bias. 2019 [cited 2021 Oct 5]. https://catalogofbias.org/biases/industry-sponsorship-bias/

  63. Bankhead C, Spencer E, Nunan D. Information bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/information-bias/. Accessed 5 Oct 2021.

  64. Heneghan C, Brassey J. Insensitive measure bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/insensitive-measure-bias/. Accessed 5 Oct 2021.

  65. Nunan D, Heneghan C. Lack of blinding. Catalogue of Bias. 2018. https://catalogofbias.org/biases/lack-of-blinding/. Accessed 5 Oct 2021.

  66. Brassey J, Spencer E, Heneghan C. Language bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/language-bias/. Accessed 5 Oct 2021.

  67. Spencer E, Mahtani K, Brassey J, Heneghan C. Misclassification bias. Catalogue of Bias. 2018. https://catalogofbias.org/biases/misclassification-bias/. Accessed 5 Oct 2021.

  68. Spencer E, Brassey J, Heneghan C. Non-contemporaneous control bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/non-contemporaneous-control-bias/. Accessed 5 Oct 2021.

  69. Turk A, Heneghan C, Nunan D. Non-response bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/non-response-bias/. Accessed 5 Oct 2021.

  70. Persaud N, Heneghan C. Novelty bias. Catalogue of Bias. 2021. https://catalogofbias.org/biases/novelty-bias/. Accessed 5 Oct 2021.

  71. Spencer E, Brassey J, Heneghan C. One-sided reference bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/one-sided-reference-bias/. Accessed 5 Oct 2021.

  72. Thomas E, Heneghan C. Outcome reporting bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/outcome-reporting-bias/. Accessed 5 Oct 2021.

  73. Spencer E, Brassey J. Perception bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/perception-bias. Accessed 5 Oct 2021.

  74. Banerjee A, Pluddemann A, O’Sullivan J, Nunan D. Performance bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/performance-bias/. Accessed 5 Oct 2021.

  75. Plüddemann A, Banerjee A, O’Sullivan J. Positive results bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/positive-results-bias/. Accessed 5 Oct 2021.

  76. Spencer E, Heneghan C. Prevalence-incidence (Neyman) bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/prevalence-incidence-neyman-bias/. Accessed 5 Oct 2021.

  77. Devito N, Goldacre B. Publication bias. Catalogue of Bias. 2019.

  78. Spencer E, Brassey J, Mahtani K. Recall bias. 2017 [cited 2021 Oct 5]. https://catalogofbias.org/biases/recall-bias/. Accessed 5 Oct 2021.

  79. Richards G, Onakpoya I. Reporting biases. Catalogue of Bias. 2019. https://catalogofbias.org/biases/reporting-biases/. Accessed 5 Oct 2021.

  80. Nunan D, Bankhead C, Aronson J. Selection bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/selection-bias/. Accessed 5 Oct 2021.

  81. Mahtani K, Chalmers I, Nunan D. Spin bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/spin-bias/. Accessed 5 Oct 2021.

  82. Heneghan C. Starting time bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/starting-time-bias. Accessed 5 Oct 2021.

  83. Heneghan C, Spencer E. Substitution game bias. Catalogue of Bias. 2019. https://catalogofbias.org/biases/substitution-game-bias/. Accessed 5 Oct 2021.

  84. Brassey J, Mahtani K, Spencer E, Heneghan C. Volunteer bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/volunteer-bias/. Accessed 5 Oct 2021.

  85. Spencer E, Brassey J, Mahtani K, Heneghan C. Wrong sample size bias. Catalogue of Bias. 2017. https://catalogofbias.org/biases/wrong-sample-size-bias/. Accessed 5 Oct 2021.

  86. Bero L, Chartres N, Diong J, Fabbri A, Ghersi D, Lam J, et al. The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures. Syst Rev. 2018;7:242.

    Article  PubMed  PubMed Central  Google Scholar 

  87. Hudson P, Botzen WJW, Kreibich H, Bubeck P, Aerts JCJH. Evaluating the effectiveness of flood damage mitigation measures by the application of propensity score matching. Nat Hazard. 2014;14:1731–47.

    Article  Google Scholar 

  88. Takeshita KM, Hayashi TI, Yokomizo H. The effect of intervention in nickel concentrations on benthic macroinvertebrates: a case study of statistical causal inference in ecotoxicology. Environ Pollut. 2020;265: 115059.

    Article  CAS  PubMed  Google Scholar 

  89. Konno K, Akasaka M, Koshida C, Katayama N, Osada N, Spake R, et al. Ignoring non-English-language studies may bias ecological meta-analyses. Ecol Evol. 2020;10:6373–84.

    Article  PubMed  PubMed Central  Google Scholar 

  90. Konno K, Pullin AS. Assessing the risk of bias in choice of search sources for environmental meta-analyses. Res Synth Methods. 2020. https://doi.org/10.1002/jrsm.1433.

    Article  PubMed  Google Scholar 

  91. Pullin AS, Cheng SH, Jackson JD, Eales J, Envall I, Fada SJ, et al. Standards of conduct and reporting in evidence syntheses that could inform environmental policy and management decisions. Environ Evid. 2022;11:16.

    Article  Google Scholar 

  92. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.

    Article  CAS  PubMed  Google Scholar 

  93. Pullin AS, Stewart G. Guidelines for systematic review in conservation and environmental management. Conserv Biol. 2006;20:1647–56.

    Article  PubMed  Google Scholar 

  94. James KL, Randall NP, Haddaway NR. A methodology for systematic mapping in environmental sciences. Environ Evid. 2016;5:7. https://doi.org/10.1186/s13750-016-0059-6.

    Article  Google Scholar 

  95. Minozzi S, Cinquini M, Gianola S, Gonzalez-Lorenzo M, Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol. 2020;126:37–44.

    Article  PubMed  Google Scholar 

  96. Minozzi S, Cinquini M, Gianola S, Castellini G, Gerardi C, Banzi R. Risk of bias in nonrandomized studies of interventions showed low inter-rater reliability and challenges in its application. J Clin Epidemiol. 2019;112:28–34.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

We did not receive any funding support.

Author information

Authors and Affiliations

Authors

Contributions

KK: Conceptualization (formulation of overarching research goals and aims), Methodology (development of methodology), Investigation (conducting a research and investigation process), Data Curation (management activities to annotate (produce metadata) and maintain research data for initial use and later reuse), Writing—Original Draft, Visualization (creating figures and tables); JG: Supervision, Writing—Review & Editing; RL: Supervision, Writing—Review & Editing; ASP: Supervision, Writing—Review & Editing.

Corresponding author

Correspondence to Ko Konno.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

 Excluded types of bias.

Additional file 2.

 Included types of bias.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Konno, K., Gibbons, J., Lewis, R. et al. Potential types of bias when estimating causal effects in environmental research and how to interpret them. Environ Evid 13, 1 (2024). https://doi.org/10.1186/s13750-024-00324-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13750-024-00324-7

Keywords