Potential types of bias when estimating causal effects in environmental research and how to interpret them

Abstract

To inform environmental policy and practice, researchers estimate effects of interventions/exposures by conducting primary research (e.g., impact evaluations) or secondary research (e.g., evidence reviews). If these estimates are derived from poorly conducted/reported research, then they could misinform policy and practice by providing biased estimates. Many types of bias have been described, especially in health and medical sciences. We aimed to map all types of bias from the literature that are relevant to estimating causal effects in the environmental sector. All the types of bias were initially identified by using the Catalogue of Bias (catalogofbias.org) and reviewing key publications (n = 11) that previously collated and described biases. We identified 121 (out of 206) types of bias that were relevant to estimating causal effects in the environmental sector. We provide a general interpretation of every relevant type of bias covered by seven risk-of-bias domains for primary research: risk of confounding biases; risk of post-intervention/exposure selection biases; risk of misclassified/mismeasured comparison biases; risk of performance biases; risk of detection biases; risk of outcome reporting biases; risk of outcome assessment biases, and four domains for secondary research: risk of searching biases; risk of screening biases; risk of study appraisal and data coding/extraction biases; risk of data synthesis biases. Our collation should help scientists and decision makers in the environmental sector be better aware of the nature of bias in estimation of causal effects. Future research is needed to formalise the definitions of the collated types of bias such as through decomposition using mathematical formulae.

Background

Estimation of average causal effects of interventions or exposures can provide important evidence to inform environmental policy and practice. There is a risk that if these estimates are derived from poorly conducted (or selective reporting or selectively reported [1,2,3,4]) primary (e.g., impact evaluation studies) or secondary research (e.g., evidence reviews), then they will potentially misinform policy and practice. Biased estimates of average causal effects (inaccuracy) are more problematic than imprecise estimates of average causal effects due to random error which may be measured by standard errors or confidence intervals [5] because a bias is a systematic error or ‘deviation from the truth’ [6, 7], and can be a direct threat to the validity of a causal inference [8]. Biases can be quantified only when the true average causal effects are known, or when hypothetical true (or expected) average effects are set as reference values (i.e., B = m − µ, where m is an estimated average causal effect and µ is the true causal effect [7, 9, 10]; it is also defined as $E\left( {\widehat{\theta }} \right) - \theta$ [8]). This means that, when there is a systematic error, an estimated average causal effect is inaccurate and either an under- or overestimate of the true average causal effect (i.e., m = µ ± B, where B ≠ 0 [10]). However, distinguishing biases from random errors and quantification of biases may not be possible using real-world data [6]. Instead of quantifying biases, the extent of ‘risk of bias’ or ‘threats to internal validity’ may need to be communicated explicitly [11,12,13].

Many types of bias (specific phenomena resulting in systematic errors described in the literature and on the Internet) have been described, most notably in medical and health sciences where methods for assessing the risk of bias originated (e.g., Catalogue of Bias describes 62 types of bias as of 29 August 2021: catalogofbias.org). For example, observer bias is described as ‘the process of observing and recording information which includes systematic discrepancies from the truth’ [14], and detection bias is described as ‘systematic differences between groups in how outcomes are determined’ [15]. Types of bias, which are considered by individuals or groups as distinctive phenomena that can give rise to biased estimates, are generally described freely, but some can also be defined formally using mathematical languages [e.g., confounding bias and collider bias are often defined using directed acyclic graphs (DAGs)] [16, 17].

In the health sector, there are dedicated tools for assessing the risk of bias or internal validity [18, 19]. For example, Risk of Bias version 2 (RoB 2) [20] and Risk Of Bias In Non-randomised Studies—of Interventions (ROBINS-I) [21] are widely recommended domain-based tools for assessing the risk of bias in primary research (the former is for randomised controlled trials and the latter is for non-randomised intervention studies) when conducting systematic reviews in the health sector [18]. There is another widely recommended domain-based tool called Risk Of Bias In Systematic reviews (ROBIS) which is for specifically assessing the risk of bias arising from the conduct of secondary research in the health sector [18, 22]. These risk-of-bias assessment tools guide users to focus on a few ‘domains’ of bias (broad categories of bias specifically designed for assessing the risk of bias in particular situations; for example, when you plan to synthesise effect sizes from multiple studies, you would need to assess the risk of bias in primary research assessing effects by developing a comprehensive set of domains to address all potential sources of bias occurring in primary research in your subject area) [13]. Risk-of-bias assessors can make overall judgements about the extent of risk of bias for the specified domains (e.g., low risk of bias, some concerns, or high risk of bias [20]). The concept of domains of bias was initially developed in the health sector to assess the risk of bias for groups of types of bias that can occur at specific stages of primary or secondary research [20, 21]. For example, in the ROBINS-I tool, detection bias, recall bias, information bias, misclassification bias, observer bias, and measurement bias are all included in a domain called ‘risk of bias in measurement of outcome’ [21].

Following the developments of the risk-of-bias assessment tools in the health sector, the Collaboration for Environmental Evidence (hereafter ‘CEE’: environmentalevidence.org) is adopting such a domain-based approach and has been developing a domain-based critical appraisal tool called ‘CEE Critical Appraisal Tool’ since 2020. This tool is for assessing the risk of bias in primary research and is available for use in evidence reviews in the environmental sector (environmentalevidence.org/cee-critical-appraisal-tool). Employing domains of bias is now strongly encouraged when conducting evidence reviews in medical and health sciences [20, 21, 23, 24] as well as in the environmental sector [13] because assessing every individual type of bias is laborious and there are no universal definitions of all existing types of bias in estimation of causal effects in science. A recent review on critical appraisal in ecology highlighted this important development and the urgent need to evaluate the validity and reliability of the CEE Critical Appraisal Tool because, remarkably, only 4% of reviews labelled as ‘systematic reviews’ in ecology actually conducted critical appraisal [25]. This finding suggests insufficient education about biases in environmental science courses and a lack of understanding of how to avoid providing potentially biased estimates [26].

We aim to raise awareness about biases in the environmental sector by mapping types of bias that may impact the reliability of the results of environmental research evaluating the effects of interventions or exposures and categorising them into the developed domains of bias. The objectives of our research are: (1) to identify potential types of bias, that have been previously collated, in estimation of causal effects in either primary or secondary research; (2) to evaluate the relevance of the identified biases to the environmental sector, which incorporates not only non-human biotic (e.g., plants, animals) and abiotic subjects (e.g., water, atmosphere) but also human subjects; and (3) to utilise the resultant list of biases to evaluate the coverage of the domains of bias for a previously developed assessment tool to use in environmental research, namely the CEE Critical Appraisal Tool.

To avoid confusion, when we refer to domains of bias, we use plural rather than singular (e.g., confounding biases) as noted below. Also, note that we do not aim to evaluate relatedness among biases (e.g., we do not discuss whether selection bias is related to attrition bias [27]). Rather we aim to provide an overview of the types of bias from the view of managing the risk of bias by employing the developed domains.

Methods

Listing the types of bias previously collated and described elsewhere

In order to develop a comprehensive list of potential biases we used the Catalogue of Bias (catalogofbias.org), and reviewed 11 key publications in the field: Sackett [28], Bayliss and Beyer [29], Clarke et al. [30], Smith and Noble [31], Thakur et al. [32], Paradis [33], Warden [34], Delegado-Rodriguez and Llorca [35], Hartman et al. [36], Marchevsky [37], and Pannucci and Wilkins [38]. The Catalogue of Bias is an ongoing collaborative project aiming to map all the biases that affect health and medical evidence. The Catalogue of Bias platform was developed by the Centre for Evidence-Based Medicine (CEBM) at Oxford University through reviewing the literature and regular meetings. We considered the Catalogue of Bias to provide a robust and comprehensive source. To our knowledge, there was no equivalent database of bias in environmental sciences, so the Catalogue of Bias was the only choice, apart from the 11 key publications. The additional 11 key publications used as information sources were selected because they were known to the authors as seminal papers that collated and described biases. We listed biases that were described in these sources in a Microsoft Excel spreadsheet (Redmond, WA, USA). The intention here was to identify the breadth of biases that have been recognised or described and not to identify all studies describing each bias. We therefore did not conduct a systematic literature search. The broad scope of our review and the time and resources available also precluded this.

Evaluating the relevance of the biases to estimation of effects in the environmental sector

Since the vast majority of the types of bias were described in the medical and health research contexts, we evaluated them for relevance to estimation of effects of interventions or exposures in the environmental sector. Relevance was evaluated by a single reviewer (KK) using the algorithm provided in Fig. 1.

Fig. 1

The algorithm used for evaluating the relevance of biases described in the sources

We employed a broad and all-inclusive scope for evaluating relevance to the environmental sector. The following a priori list of relevant topics was developed and used for assessing relevance: agriculture, aquaculture, biodiversity conservation, climate change, ecology and evolution, ecosystem services, environment and human wellbeing, environmental education, environmental legislation, fisheries, food security, forestry, invasive species, natural resource management, park and protected area management, pollution, soil management, sustainable energy use, waste management, wastewater management, water security, as well as any research relevant to environmental sustainability. The broad scope is consistent with that of CEE. The scope inherently included not only non-human biotic and abiotic subjects but also human subjects, however, human health outcomes were out of scope. The algorithm was applied by one person (KK) and the resulting decisions were checked by a second person (AP), and any disagreements or uncertainties were discussed among all authors. We highlight that we did not carry out any consistency checking, as we did not aim to aggregate effects, and did not aim to provide databases for aggregating effects. Hence, Cohen’s kappa statistic [39, 40] was not calculated.

Evaluating the coverage of the developed domains of bias

Suzuki et al. [8] provided an organisational schema for systematic error in causal inference in primary research. In primary research, systematic error can be divided into structural error and analytic error. Structural error can be further divided into bias relating to measurement of intervention, exposure or outcome, and bias relating to exchangeability. Exchangeability refers to independence between the counterfactual outcome ${Y}^{a}$ (the outcome which a subject or area would have experienced if it had received a specified intervention or exposure value [41]) and the observed intervention or exposure $A$, and it is denoted as Y^a ⫫ A, for all treatment values a (all values of the treatment variable that may differ among individual subjects or areas) [42]. Alternatively, exchangeability refers to an assumption that the outcome in the comparator group would mirror the outcome in the intervention or exposure group if the subjects or areas in the comparator group had been subjected to the same intervention or exposure as those in the intervention or exposure group [17]. Analytic error is a systematic error that cannot be explained by structural error, and it includes bias relating to reporting [20, 21] and statistical methods [23, 24, 43]. Figure 2 illustrates our assumptions about the organisation of bias occurring in primary and secondary research from the broadest term (systematic error) to the developed domains of bias.

Fig. 2

Organisation of bias from the broadest term (systematic error) to the developed domains of bias

Frampton et al. [13] provided the principles of risk-of-bias assessment in the environmental sector and recommended seven domains (or classes) to be assessed when conducting evidence reviews assessing the effect of an intervention or exposure. These domains for primary research were consistent with the CEE Critical Appraisal Tool version 0.3 (see environmentalevidence.org/cee-critical-appraisal-tool for more details) as including the following:

• 1. Confounding biases: Risk of biases due to an uncontrolled (or inappropriately controlled) variable (confounder) that influences both the intervention/exposure and the outcome

• 2. Post-intervention/exposure selection biases: Risk of biases arising from systematic differences in the selection of subjects or areas into the study or analysis after the intervention or exposure

• 3. Misclassified/mismeasured comparison biases: Risk of biases arising from misclassification or mismeasurement of the intervention, exposure and/or comparator (applicable to observational studies only)

• 4. Performance biases: Risk of biases due to altered treatment procedure of interest (applicable to experimental studies only; treatment procedures are series of actions for applying the experimental (intervention or exposure) treatment [44]; note that an experimental treatment can be either an intervention or an exposure, or both, depending on the outcome measure. For example, when a fertilizer is applied to a crop, this is an intervention if the outcome measure is crop yield, but an exposure if the outcome measure is change in soil microfauna [45])

• 5. Detection biases: Risk of biases arising from systematic differences in measurement of outcomes

• 6. Outcome reporting biases: Risk of biases in reporting of study findings

• 7. Outcome assessment biases: Risk of biases due to error in applied statistical methods

Regarding the domains for secondary research, we used the CEE Synthesis Appraisal Tool (CEESAT) version 2 [46, 47]. CEESAT is an eight-criteria checklist consisting of sixteen questions for assessing environmental evidence reviews in terms of risk of bias, repeatability, and transparency, and hence not all the criteria are relevant to risk of bias (e.g., review question setting and provision of limitations). Using only the relevant criteria resulted in four domains of bias which were the same as the ROBIS tool [22], and thus consistent with a widely recommended tool in medical and health sciences [18]:

• 1. Searching biases: Risk of biases in searches of relevant records

• 2. Screening biases: Risk of biases arising from screening of potentially relevant records

• 3. Study appraisal and data coding/extraction biases: Risk of biases due to the lack of or inappropriate conduct of study appraisal and data coding/extraction

• 4. Data synthesis biases: Risk of biases due to employing inappropriate synthesis methods

Each type of bias was checked by a single reviewer (KK) for relevancy against each domain. Each bias-and-domain combination was data-coded in the database (spreadsheet), by the single reviewer using subjective judgements (‘Yes’ = relevant/‘No’ = not relevant). All decisions were double-checked by the other authors. The evaluator was allowed to take special notes when some types of bias were beyond the provided domains but there were no such cases. We also data-coded relevant levels of research (primary and/or secondary research) for all included types of bias.

Results

Types of bias relevant to the environmental sector

Figure 3 summarises the results of the bias selection process and provides the numbers of biases identified in each of the sources searched. Some biases were described by multiple sources while some biases were described by one source only. There were 206 types of bias after merging 53 bias descriptors that were specifically described as synonyms; we did not merge bias descriptors unless they were specifically referred to as synonyms to make sure none are missed from the list. Note that we did not reproduce (verbatim) the descriptions of the biases provided in the sources mainly due to copyright restrictions.

Fig. 3

Identification of biases previously collated and described elsewhere

After the evaluation of relevance, we identified 121 types of bias (accompanied by 24 synonyms specified in the sources) that were relevant to estimation of effects in the environmental sector. Of the 85 excluded types of biases, 53 (62%) were excluded due to a lack of relevance to estimation of effects. Thirty-one (37%) were excluded due to a lack of relevance to the environmental sector; the most common reasons for exclusions were specifically referring to human health outcomes, diagnosis, and healthcare and medical interventions. One (1%; culture bias described by Warden 2015 [34]) was excluded because its description was insufficient for judging its relevance. All the excluded types of bias can be found in Additional file 1.

The general interpretations and relevant domains of the 121 types of bias relevant to the environmental sector are provided in Table 1. Sixty-eight types of bias (56%) were relevant to primary research only and 18 types of bias (15%) were relevant to secondary research only. Thirty-five types of bias (29%) were relevant to both the primary and secondary research. We provide the database of these biases in Additional file 2 so that readers can use the filter function of the spreadsheet to find types of bias relevant to specific domains. To increase its convenience, we also created a sheet to list types of bias relevant to each domain.

Table 1 Types of bias relevant to the environmental sector in the alphabetical order

The coverage of the domains of bias

There was no type of bias that was not applicable to any domain, and we did not find any types of bias that were beyond the domains, suggesting that domains were sufficient to cover the 121 types of bias. Eighty-six types of bias (71%) were applicable to multiple domains while 35 types of bias (29%) were applicable to only one domain of bias.

Discussion

Overview of findings

We found 121 (out of 206) types of bias that were relevant to estimation of causal effects in the environmental sector. We provided a general interpretation of every relevant type of bias covered by seven risk-of-bias domains for primary research and four domains for secondary research.

Strengths and limitations

As far as we are aware this is the first study to provide a comprehensive map of the type of bias that may affect the reliability of the results of environmental research studies and reviews. It provides an important summary of the breadth of biases which may impact the findings. There are limitations that should be noted. We used a pragmatic approach to identify potentially relevant biases by utilising the Catalogue of Bias and 11 key publications. We did not conduct a systematic literature search, similar to the approach used by some of the key documents we consulted [28,29,30,31,32,33, 35,36,37,38]. However, the Catalogue of Bias which is developed by CEBM at Oxford University and is continually updated, is considered a robust source for identifying biases that affect health and medical evidence. There may be other relevant types of bias that are collated and described elsewhere, but we are not aware of any published studies specific to the environmental sector that aimed to identify types of bias. We used subjective judgements for evaluating relevance and some aspects of biases were open for interpretation although we conducted independent double checking. We provided only ‘general’ interpretations of the types of bias in this paper.

Implications for research and practice

Since education about biases is not sufficient and the level of knowledge about biases are concerning in the environmental sector [26], we hope the provision of the general interpretations of the 121 types of bias helps environmental scientists, as well as decision-makers, be better aware of the biases. When environmental scientists, practitioners, and policymakers come across certain types of bias when they read research or review papers, or when they communicate about certain types of bias, we suggest they check our dedicated list of biases relevant to estimation of causal effects in the environmental sector.

Although some types of bias such as confounding bias, selection bias, and measurement bias have been formally defined [8, 17], future research is needed to formalise definitions of other types of bias such as through decomposition using mathematical formulae and/or by employing directed acyclic graphs (DAGs). For example, Suzuki et al. 2016 [8] decomposed confounding bias by comparing the causal measure (A → Y), µ(E(Y₁), E(Y₀)), and the situation where a confounding variable is present as:

$$\mu \left( {E\left( {Y{|}A = 1} \right), E\left( {Y{|}A = 0} \right)} \right) - \mu \left( {E\left( {Y_{1} } \right),E\left( {Y_{0} } \right)} \right)$$

where Y refers to outcome and A refers to intervention or exposure (A = 1 receives the intervention or exposure and A = 0 does not receive the intervention or exposure). Although the practicality of defining all biases remains to be evaluated, formalising definitions is particularly important because, to some extent, vagueness remains in the descriptions of the biases. This would also help identify duplication and cross over between biases identified by different researchers.

As the types of bias are described theoretically, future research is also needed for collating empirical studies that evaluate the impacts of the identified biases. Such research is desired even in the health sector for better managing the risk of bias [86]. In the environmental sector, for example, Hudson et al. [87] and Takeshita et al. [88] evaluated the potential extent of selection bias and confounding bias, respectively. Konno et al. [89] and Konno and Pullin [90] evaluated the potential extent of language bias and availability bias, respectively. If such studies are collated and summarised, environmental scientists should be better able to manage the risk of bias. We hope this paper guides future research.

A lack of and inappropriate conduct of risk-of-bias assessment in the environmental sector has been reported in the literature [13, 25, 91]. Our findings showed that the domains selected for use in CEE Critical Appraisal Tool and CEE Synthesis Assessment Tool are sufficient and cover all the biases identified in primary and secondary research (Fig. 2) and can be used to facilitate communication about the risk of bias. A domain-based tool should incorporate a series of checklist questions within each domain that prompts users to consider relevant types and sources of bias. Developing a series of checklist questions requires relevant expertise [20, 21], and thus, we encourage evidence providers (e.g., impact evaluators, evidence synthesists) as well as evidence consumers (e.g., decision-makers) to make use of the existing tools for assessing the risk of bias (e.g., the CEE Critical Appraisal Tool for primary research and the CEE Synthesis Appraisal Tool for secondary research), rather than developing a tool on their own, as it may result in inappropriate conduct of risk-of-bias assessment [13, 91]. However, at the time of writing, the CEE Critical Appraisal Tool is still under development, and we hope users of this tool will provide their feedback for the developers. The most important aspect for further development of the tool is inter-rater reliability (i.e., consistency between assessors) which can be quantified using a kappa statistic [92] and reported in eventual publications (e.g., Systematic Reviews [93], Systematic Maps [94]). Even in medical and health sciences, inter-rater reliability is still an issue [95, 96].

As part of our future work, we plan to consult researchers and policymakers to reach a consensus on the list of biases for better and active communication within the communities. We highlight the importance of including appropriately qualified and experienced stakeholders in any consultation because it is likely that many people working in environmental research and policy are inadequately familiar with biases [26], and thus there is a need to educate environmental scientists and decision-makers about the types of bias. There is also a need to provide training and develop capacity for use of risk-of-bias assessment tools in the environmental sciences. We are not aware of any institution that teaches this subject in environmental contexts, even at Masters level. In response to this gap, CEE intends to develop a series of online training videos on critical appraisal. If many more scholars, institutions as well as learned societies can help the environmental sector better understand the nature of biases, more active communication of the risk of bias may be achieved.