Table 2 Criteria for the assessment of internal validity of a study, using the Environmental-Risk of Bias Tool, adapted from the Cochrane Collaboration’s Risk of Bias Tool

SELECTION BIAS DUE TO INADEQUATE RANDOMISATION

Selection bias refers to systematic differences between baseline characteristics of the study units that are to be compared [66]. The likelihood of selection bias can be minimised through randomisation of treatments to study units and through allocation sequence concealment [66]. Proper randomisation requires that a rule for allocating treatments to study units must be specified based on some chance (random) process [66]. This is referred to as the sequence generation [66].

Criteria for an assessment of ‘Low risk’ of bias.

The investigators describe a random component in the sequence generation process such as:

• Referring to a random number table;

• Using a computer random number generator;

• Coin tossing;

• Shuffling cards or envelopes;

• Throwing dice;

• Drawing of lots;

• Minimisation¹

Criteria for an assessment of ‘High risk’ of bias.

The investigators describe a non-random component in the sequence generation process. Usually, the description would involve some systematic, non-random approach, for example:

• Sequence generated by odd or even date of birth of study units, or odd or even latitude of sites;

• Sequence generated by some rule based on site number or site code in a database;

Other non-random approaches involving judgement or some method of non-random allocation of study units, for example:

• Allocation by judgement of the investigator;

• Allocation based on the results of a laboratory test or a series of tests;

• Allocation by availability of the intervention.

Criteria for an assessment of ‘Unclear risk’ of bias.

Insufficient information about the sequence generation process to permit assessment of ‘Low risk’ or ‘High risk’.

SELECTION BIAS DUE TO INADEQUATE ALLOCATION CONCEALMENT

Selection bias refers to systematic differences between baseline characteristics of the study units that are to be compared [66]. The likelihood of selection bias can be minimised through randomisation of treatments to study units and through allocation sequence concealment [66]. Proper allocation concealment involves taking steps to secure strict implementation of that schedule of random assignments by preventing foreknowledge (by study units and investigators) of the forthcoming allocations [66]. This process if often termed allocation concealment, although it could more accurately be described as allocation sequence concealment [66].

Criteria for an assessment of ‘Low risk’ of bias.

Study units and investigators enrolling study units could not foresee allocations because one of the following, or an equivalent method, was used to conceal allocation:

• Central allocation (including telephone, web-based and database controlled randomisation);

• Sequentially numbered treatment containers of identical appearance (e.g. test of effectiveness of different pesticides, where different pesticides appear identical);

Criteria for an assessment of ‘High risk’ of bias.

Study units or investigators enrolling study units could possibly foresee allocations and thus introduce selection bias, such as allocation based on:

• Using an open random allocation schedule (e.g. a list of random numbers);

• Alternation or rotation;

• Date of birth of participants;

• Case record number/site code;

• Any other explicitly unconcealed procedure.

Criteria for an assessment of ‘Unclear risk’ of bias.

Insufficient information to permit assessment of ‘Low risk’ or ‘High risk’. This is usually the case if the method of concealment is not described or not described in sufficient detail to allow a definite assessment.

PERFORMANCE BIAS

Performance bias refers to systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest [66]. After enrolment into the study, blinding (or masking) of study units and personnel (assuming these are different to the investigators responsible for allocation of study units to study groups) may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcomes [66]. Effective blinding can also ensure that the compared groups receive a similar amount of attention, ancillary treatment and diagnostic investigations [66].

Criteria for an assessment of ‘Low risk’ of bias.

Any one of the following:

• No blinding or incomplete blinding, but the review authors judge that the outcome is not likely to be influenced by lack of blinding;

• Blinding of study units and key study personnel ensured, and unlikely that the blinding could have been broken.

Criteria for an assessment of ‘High risk’ of bias.

Any one of the following:

• No blinding or incomplete blinding, and the outcome is likely to be influenced by lack of blinding;

• Blinding of key study units and personnel attempted, but likely that the blinding could have been broken, and the outcome is likely to be influenced by lack of blinding.

Criteria for an assessment of ‘Unclear risk’ of bias.

.

Any one of the following:

• Insufficient information to permit assessment of ‘Low risk’ or ‘High risk’;

The study did not address this outcome

DETECTION BIAS

Detection bias refers to systematic differences between groups in how outcomes are determined [66]. Blinding of outcome assessors may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcome measurement [66]. Blinding of outcome assessors can be especially important for assessment of subjective outcomes [66].

Criteria for an assessment of ‘Low risk’ of bias.

Any one of the following:

• No blinding of outcome assessment, but the review authors judge that the outcome measurement is not likely to be influenced by lack of blinding;

• Blinding of outcome assessment ensured, and unlikely that the blinding could have been broken.

Criteria for an assessment of ‘High risk’ of bias.

Any one of the following:

• No blinding of outcome assessment, and the outcome measurement is likely to be influenced by lack of blinding;

• Blinding of outcome assessment, but likely that the blinding could have been broken, and the outcome measurement is likely to be influenced by lack of blinding.

Criteria for an assessment of ‘Unclear risk’ of bias.

Any one of the following:

• Insufficient information to permit assessment of ‘Low risk’ or ‘High risk’;

The study did not address this outcome.

ATTRITION BIAS DUE TO INCOMPLETE OUTCOME DATA

Attrition bias refers to systematic differences between groups in withdrawals from a study [66]. Withdrawals from the study lead to incomplete outcome data [66]. There are two reasons for withdrawals or incomplete outcome data in clinical trials [66]. Exclusions refer to situations in which some study units are omitted from reports of analyses, despite outcome data being available to the trialists [66]. Attrition refers to situations in which outcome data are not available [66].

Criteria for an assessment of ‘Low risk’ of bias.

Any one of the following:

• No missing outcome data;

• Reasons for missing outcome data unlikely to be related to true outcome (for survival data, censoring unlikely to be introducing bias);

• Missing outcome data balanced in numbers across intervention groups, with similar reasons for missing data across groups;

• For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk not enough to have a relevant impact on the intervention effect estimate;

• For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes not enough to have a relevant impact on observed effect size;

• Missing data have been imputed using appropriate methods.

Criteria for an assessment of ‘High risk’ of bias.

Any one of the following:

• Reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across intervention groups;

• For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk enough to induce relevant bias in intervention effect estimate;

• For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes enough to induce relevant bias in observed effect size;

• ‘As-treated’ analysis done with substantial departure of the intervention received from that assigned at randomization;

• Potentially inappropriate application of simple imputation.

Criteria for an assessment of ‘Unclear risk’ of bias.

Any one of the following:

• Insufficient reporting of attrition/exclusions to permit assessment of ‘Low risk’ or ‘High risk’ (e.g. number randomized not stated, no reasons for missing data provided);

The study did not address this outcome.

REPORTING BIAS DUE TO SELECTIVE REPORTING

Reporting bias refers to systematic differences between reported and unreported findings [66]. Within a published report those analyses with statistically significant differences between treatment groups are more likely to be reported than non-significant differences [66]. This sort of ‘within-study publication bias’ is usually known as outcome reporting bias or selective reporting bias, and may be one of the most substantial biases affecting results from individual studies [67].

Criteria for an assessment of ‘Low risk’ of bias.

Any of the following:

• The study protocol is available and all of the study’s pre-specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre-specified way;

• The study protocol is not available but it is clear that the published reports include all expected outcomes, including those that were pre-specified (convincing text of this nature may be uncommon).

Criteria for an assessment of ‘High risk’ of bias.

Any one of the following:

• Not all of the study’s pre-specified primary outcomes have been reported;

• One or more primary outcomes is reported using measurements, analysis methods or subsets of the data (e.g. subscales) that were not pre-specified;

• One or more reported primary outcomes were not pre-specified (unless clear justification for their reporting is provided, such as an unexpected adverse effect);

• One or more outcomes of interest in the review are reported incompletely so that they cannot be entered in a meta-analysis;

• The study report fails to include results for a key outcome that would be expected to have been reported for such a study.

Criteria for an assessment of ‘Unclear risk’ of bias.

Insufficient information to permit assessment of ‘Low risk’ or ‘High risk’. It is likely that the majority of studies will fall into this category.

OTHER BIAS

In addition there are other sources of bias that are relevant only in certain circumstances [66]. These relate mainly to particular study designs (e.g. carry-over in cross-over trials and recruitment bias in cluster-randomized trials); some can be found across a broad spectrum of trials, but only for specific circumstances (e.g. contamination, whereby the experimental and control interventions get ‘mixed’); and there may be sources of bias that are only found in a particular setting [66].

Criteria for an assessment of ‘Low risk’ of bias.

The study appears to be free of other sources of bias.

Criteria for an assessment of ‘High risk’ of bias.

There is at least one important risk of bias. For example, the study:

• Had a potential source of bias related to the specific study design used; or

• Has been claimed to have been fraudulent; or

• Had some other problem.

Criteria for an assessment of ‘Unclear risk’ of bias.

There may be a risk of bias, but there is either:

• Insufficient information to assess whether an important risk of bias exists; or

• Insufficient rationale or evidence that an identified problem will introduce bias.