Open Access

Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of non-GM maize? A systematic review protocol

  • Michael Meissle1Email author,
  • Steven E Naranjo2,
  • Christian Kohl3,
  • Judith Riedel1 and
  • Jörg Romeis1
Environmental EvidenceThe official journal of the Collaboration for Environmental Evidence20143:7

DOI: 10.1186/2047-2382-3-7

Received: 28 January 2014

Accepted: 13 April 2014

Published: 24 April 2014



Since 1996, genetically modified (GM) crops have been grown on an ever increasing area worldwide. Maize producing a Cry protein from the bacterium Bacillus thuringiensis (Bt) was among the first GM crops released for commercial production and it is the only GM crop currently cultivated in Europe. A major part of the regulatory process that precedes the commercial release of GM crops is the environmental risk assessment. Because Bt maize is modified to produce insecticidal proteins, potential interactions with non-target organisms are a major area of concern to be addressed in the risk assessment. In particular, beneficial arthropods that provide important agro-ecological services, such as pollination, decomposition, and biological control are the focus. This systematic review will evaluate if the growing of Bt maize changes abundance or ecological function of non-target animals compared to the growing of conventional, non-GM maize. The review will be limited to plot or field level data including field margins. Potential cropping system effects and off-field effects will not be addressed. Bt maize will be compared to conventional maize either untreated or treated with chemical insecticides.


Stakeholders from academia, competent authorities, industry, and civil society organizations were given the opportunity to comment on the review question and an earlier draft of this review protocol.

Keyword searches will be conducted in a range of abstracting and full text literature databases. Retrieved records will be screened against a set of inclusion criteria, first on title and abstract level, then on full text level. Selected studies will be evaluated for risks of bias (quality assessment). Data on field/plot characteristics, maize cultivars, insecticide treatments, non-target animal taxa, sampling methods, and response variables of populations and ecological functions will be extracted. Meta-analysis will be conducted using the effect size estimator Hedge’s d on a range of comparisons and including sensitivity analysis. The review process will be fully documented in CADIMA, an open access online data portal for evidence synthesis.


Agriculture Biotechnology Bt crops Corn Cry proteins Fauna Genetically modified plants Insect-resistant plants Meta-analysis Systematic review


Since 1996, genetically modified (GM) crops have been grown on an ever increasing area worldwide, reaching 170 million hectares in 2012 [1]. This area represents more than 10% of the global cultivation area of arable crops [2]. The GM maize event MON810, targeting corn borers (Lepidoptera) and producing Cry1Ab from the bacterium Bacillus thuringiensis (Bt), has been cultivated in Europe on significant areas for more than a decade. In 2012, it was planted on a total area of 129,000 ha in 5 European countries led by Spain [1, 3]. Before new GM plants can be released into the environment, regulatory approval needs to be obtained. While MON810 is the only transformation event currently approved for commercial cultivation in the EU, applications for events expressing Cry3 proteins targeting the corn rootworm (Coleoptera) as well as stacked events (producing several Bt proteins simultaneously) have been submitted [4]. A major part of the regulatory process is the environmental risk assessment. Typical environmental protection goals are farmland biodiversity conservation and sustainable land use by maintaining agro-ecological functions (functional biodiversity) [57]. Typical environmental concerns related to those protection goals are a decline in biodiversity and a disruption of agro-ecological functions. This may ultimately lead to yield reduction (Figure  1). For Europe, the European Food Safety Authority (EFSA) lists the following specific areas of risk to be addressed in the environmental risk assessment [8]: (1) Persistence and invasiveness, including plant-to-plant gene flow; (2) Potential for plant to micro-organisms gene transfer; (3) Interaction of the GM plant with target organisms; (4) Interaction of the GM plant with non-target organisms; (5) Impacts of the specific cultivation, management and harvesting techniques; (6) Effects on biochemical processes; and (7) Effects on human and animal health (Figure  1).
Figure 1

Conceptual model of interactions of the cultivation of Bt maize with non-target organisms including the potential pathways to harm (hazard, exposure, risk characterization), typical environmental concerns, and typical environmental protection goals. The planned systematic review will cover population-level effects on non-target animals, on biodiversity, and on agro-ecological functions (green box).

Because Bt maize is modified to produce insecticidal proteins, potential interactions with non-target organisms are a major area of concern to be addressed in the risk assessment. One focus of the assessment is on beneficial arthropods that provide important agro-ecological services, such as pollination, decomposition, and biological control (Figure  1). There are three plausible pathways how those organisms may be harmed: (i) the production of Bt proteins may lead to direct adverse effects on non-target animals; (ii) transformation-related, unintended changes in plant composition of nutrients or toxicants may lead to adverse effects on non-target animals; and (iii) Bt proteins and/or transformation-related effects may influence herbivorous species which triggers adverse food web effects. Common to all pathways is that changes of field populations of non-target species are a prerequisite for harm. This systematic review thus focuses on populations of non-target animals in Bt maize fields (Figure  1). Bt maize often replaces conventional pest control systems that include the application of chemical insecticides. Therefore, pest control with Bt maize will be compared to both insecticide-treated and untreated conventional fields. However, the assessment of impacts of cultivation, management, and harvesting techniques is not the focus of this review and will not be addressed in more depth (even though they might need to be addressed in the environmental risk assessment in some jurisdictions including the EU, see Figure  1).

A range of biosafety research programmes have been conducted in European countries such as Spain, Germany, Czech Republic, Hungary, Poland, and Romania. Usually, those programmes included experimental field studies on potential effects of Bt maize on non-target arthropods [9] and many data have become available in the last few years. Those studies include data on Lepidoptera-active and Coleoptera-active Bt maize. Scientists in the USA have conducted meta-analyses on global field data on the impact of Bt crops on non-target invertebrates published between 1992 and early 2008 [1013]. The current project builds on the previous meta-analyses and incorporates datasets published after 2008. With a broader scope, the planned review will also consider datasets not included in previous meta-analyses. The objective of the review is to answer the following question:

Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of conventional, non-GM maize?

The following components are contained in the question:

  •  Population: Animals (invertebrates and vertebrates) recorded in maize crops and their margins except the targets of the expressed Bt proteins: Ostrinia nubilalis, Sesamia nonagrioides, and Helicoverpa zea for Cry1, S. nonagrioides and H. zea for Vip3, and Diabrotica spp. for Cry3 proteins.

  •  Intervention: Growing of maize cultivars genetically modified to produce insecticidal proteins derived from Bacillus thuringiensis, including Cry and Vip proteins.

  •  Comparator: Growing of conventional maize either untreated or treated with insecticides. The comparison of conventional to Bt maize is possible in different spatial scales ranging from an experimental plot design within one field to a landscape study including several commercial Bt and conventional fields.

  •  Outcome: Effects of the Bt maize on some aspects of life history, abundance, or behavior of non-target animals, their function (e.g. parasitization, predation, pollination, decomposition), or their species richness.

The review will be limited to plot-to-plot and field-to-field comparisons (including field margins). Potential effects on the whole cropping system (crop rotation practices, landscape-level changes, etc.) and off-field effects will not be addressed.

With the systematic review, a broad overview of the available data (and potential data gaps) will become available. In particular, it will identify

  •  from which countries/continents data are available

  •  which taxonomic/functional groups of animals have been studied

  •  which parameters have been measured in field studies

  •  which Bt proteins were expressed in the plants

This review is one activity of the EU-funded project GRACE (G MO R isk A ssessment and C ommunication of E vidence), which aims to provide comprehensive reviews of the evidence on the health, environmental and socio-economic impacts of GM plants – considering both risks and possible benefits ( Active stakeholder involvement is a key element of GRACE to guarantee a broader acceptance of the conducted reviews and to ensure their thoroughness and their relevance from a societal perspective. In a workshop, stakeholders were representing 11 academic institutions, 16 competent authorities, 5 civil society organizations, 1 private company, and 1 industry organisation. All stakeholders were invited to comment on the planned review questions and their prioritization as well as on the draft review protocols. According to the received comments, the initial question of this review was reformulated to be open-ended ("change" instead of "decline") and broader ("animals" instead of "arthropods"). In general, stakeholders rated the proposed review question as highly important (4.25 of 5 points). They also indicated that there is medium expert disagreement on the question (3.13 of 5) and that the question is subject to medium public awareness (3.59 of 5). For more details on the stakeholder consultation process see [14, 15]. In the further course of the review process, stakeholders will also be consulted for discussion of the review results.


Data storage

The review process will be documented in the open access database CADIMA, a web-based data portal for evidence synthesis that is under development ( CADIMA will provide purpose-built online forms for each step of the evidence synthesis process, including both protocols and reviews. CADIMA will allow data exchange with bibliography software (RIS files) and Microsoft Excel for further processing (e.g. for statistical analyses).

Search strategy

Aim of the search

The aim of the literature search is to retrieve as many relevant datasets as possible and in the most unbiased way possible. Because keywords are not standardized in environmental sciences, a high-sensitivity and low-specificity approach will be applied to capture a high proportion of existing datasets. This means that the keywords will be broad. We will search abstracting databases as well as full text databases.

Search strings

Search strings used in the abstracting literature databases (e.g. Web of Science, CAB Abstracts) will be composed of 3 mandatory parts (connected with AND), which are all focused on the "intervention" part of the review question. Terms within one part are linked with "OR". Asterisks will be used to include plural terms when this is required by the database. Quotation marks will be used for multi-word terms (e.g. "Bacillus thuringiensis").

Part 1: (maize OR corn OR "Zea mays") Limits the query to the crop for which this review is conducted.

Part 2: (field* OR plot* OR location* OR trial* OR farm-scale OR scouting OR trap* OR sampl* OR monitor*) Limits data to field studies. Because the term "field" is not always present in titles and abstracts, alternative terms, which are often linked to field studies, such as location, plot, or farm-scale, are also used. In addition, the terms scouting or trap refer to methods that are mainly used in field studies.

Part 3: (transgenic OR Bt OR "Bacillus thuringiensis" OR GM OR "genetically modified" OR "genetically engineered" OR "Cry*" OR "Vip*") Returns data on genetically modified plants.

When using more general search engines (e.g. Google Scholar), a fourth part of the search string (referring to the "population" part of the review question) will be added to get more precise hits:

Part 4: ("non-target" or "nontarget" or "natural enemy" or predator or parasitoid or decomposer or pollinator)

Primary focus of the searches will be information available in English (at least on abstract level). Additional searches in other languages (e.g. Spanish, French, German) will also be conducted. Non-English searches will be restricted to search engines that contain non-English information (e.g. Google Scholar). One example in German might be:

Mais AND Feld AND Bt AND Nichtziel* The actual format of the search strings will be adapted to meet the rules of each database individually. The exact search strings and details of how each database was searched will be recorded and provided.

Sources of information

a) Databases to be searched The following abstracting literature databases will be searched:

  •  ISI Web of Science (WOS) (1900 – 2014) (Thomson Reuters, New York, USA), contains most peer reviewed scientific publications in English language (provided by Web of Knowledge, Thomson Reuters,

  •  BIOSIS (1994–2014) (Thomson Reuters, New York, USA), comprehensive reference database for life sciences (provided by Web of Knowledge).

  •  Zoological Records (1994–2014) (Thomson Reuters, New York, USA), covers all aspects of animal research (provided by Web of Knowledge).

  •  CAB Abstracts (1984–2014) (CABI, Wallingford, UK), comprehensive database that also includes more local and non-English publications, such as regional plant protection journals (provided by OvidSP, Wolters Kluwer Health, New York,

  •  AGRICOLA (1970–2013) (National Agricultural Library, U.S. Department of Agriculture, Beltsville, USA), contains bibliographic records of materials acquired by the National Agricultural Library and cooperating institutions in agricultural and related sciences (provided by OvidSP).

  •  AGRIS (1975–2013) (Food and Agriculture Organization of the United Nations FAO, Rome, Italy), the International System for the Agricultural Sciences and Technology is an international bibliographic database of national, intergovernmental or international centers (provided by OvidSP).

  •  ProQuest Dissertations & Thesis A&I, collection of dissertations and theses from around the world (

  •  BASE Bielefeld Academic Search Engine, a voluminous search engine especially for academic open access web resources, operated by Bielefeld University Library (

General full text search engines:

For those search engines the first 200 hits will be checked.

b) Specialist searches Specialist searches will be conducted on the following webpages and web-based databases containing information specifically on environmental effects of GMOs:

c) Bt crops database The Bt crops database generated by Marvier et al. [10] ( and Naranjo [12] as well as an updated version provided by Steven E. Naranjo (personal communication) will be used as a basis for the current work.

d) References in previous reviews Review papers and meta-analyses identified during the literature screening process will be checked for potentially useful additional references. A list of review papers that contain relevant references that have not been identified through our literature searches will be provided.

e) Direct contacts In cases where field studies on non-target animals have been identified but data have not been published at all or not in a form usable for the database, corresponding scientists will be contacted and asked to provide data.

f) Non-English searches Full-text databases, such as Google Scholar are also suitable for non-English searches (e.g. German). In addition, potentially useful non-English databases might be used.

Scoping exercise

Systematic reviewing aims to search the literature in a comprehensive and transparent way. While authors of traditional reviews may also search the literature, they usually do not provide records of search terms and searched databases. The most comprehensive traditional review (up to 2009) is the meta-analysis on non-target effects of Bt crops on invertebrates by Naranjo [12], which covers 40 publications on Bt maize. We performed a scoping exercise on 22 February 2013 to check if the search terms suggested for the systematic review are suitable to retrieve the already known literature on this topic. The set of 40 papers was used as a test library against which the new searches were checked. We searched in WOS, CAB Abstracts, Agricola, and Agris with the following search string:

(maize or corn) AND (field* or plot* or location* or trial* or farm-scale or scouting or trap*) AND (transgenic or Bt or "Bacillus thuringiensis" or GM or "genetically modified").

All obtained references were checked for the presence of the references from the test library. The results are listed in Table  1. CAB abstracts delivered the best result. One paper was not found by CAB abstracts, but in both WOS and Agricola. Agris only found 2 papers. All databases combined returned 38 of 40 papers. The 2 papers not obtained were

  •  Bhatti M et al. Ecological assessment for non-target organisms in the plots of corn rootworm insect-protected corn hybrid containing MON 863 Event: 2000–2001. MRID 457916–01; 2002. p. 1–143 (This is an unpublished industry study that is not indexed in the databases searched).

  •  Lang A et al. Monitoring of the environmental effects of the Bt gene. Research project sponsored by the Bavarian State Ministry for Environment, Health and Consumer Protection. Institute for Plant Protection, Bavarian State Research Center for Agriculture; Freising-Weihenstephan, Germany; 2005. p 1–111 (This is a published institute report in German that is not indexed in the databases searched).

Table 1

Literature scoping exercise



CAB abstracts



All combined

Total hits






Papers found






Number of hits and references from Naranjo [12] (in total 40 references concerning maize only) found with different literature databases (22 February 2013).

This scoping exercise demonstrated that the selected search string is suitable to return the vast majority of the literature that was used in the previous meta-analysis and that is indexed in the large literature databases such as WOS, CAB abstracts, and Agricola. Data that are not found with literature searches (similar to the two studies mentioned previously) might be found when checking the references of previous reviews and the websites and databases on non-target effects of GM crops listed in the section on specialized searches.

Screening of articles

Study inclusion criteria

For the systematic review, articles will be included if fulfilling the following criteria:


  •  Publication includes original data. Reviews, summaries, abstracts, comment papers, conference proceedings, etc. will only be used if the same data are not available in a peer-reviewed original research paper. Each dataset will be included only once.


  •  Non-target animals inhabiting fields (including field margins) are studied (invertebrates, vertebrates), excluding studies exclusively on the target pests O. nubilalis, S. nonagrioides, and H. zea for maize expressing Cry1, S. nonagrioides and H. zea for Vip, and Diabrotica spp. for Cry3 proteins.


  •  Study on maize genetically modified to express one or more proteins from Bacillus thuringiensis (Cry or Vip proteins).

  •  Growing of Bt maize under open field conditions (not in laboratory or glasshouse)


  •  Presence of a non-Bt maize control treatment to which Bt maize is compared. Bt maize not treated with insecticides will be compared to non-Bt maize either treated or untreated and Bt maize treated with insecticides will be compared to treated non-Bt maize.


  •  Effects of Bt maize on some aspects of life history, abundance, or behaviour of non-target animals, their ecological functions (e.g. parasitization, predation, pollination, decomposition), or their species richness relative to a non-Bt control are measured.

Screening articles: applying inclusion criteria

The hits retrieved by the literature searches described previously will be transferred to Endnote X5 (Thomson Reuters). One Endnote file will be established for each search engine to ensure transparency and repeatability of the search. Subsequently, the hits of all searches will be combined into one database and duplicates will be eliminated automatically. The resulting simplified database will be checked manually. In the first instance, the inclusion criteria will be applied on titles and abstracts to remove spurious citations. At the end of this stage, one database with all remaining potentially useful citations will be created in CADIMA.

For articles remaining in this selection, full texts will be organized and then filtered further according to the inclusion criteria. At this stage, each exclusion of a study for the systematic review will be documented with the reason for exclusion.

Screening articles: quality assurance process

At the beginning of the screening process, a random subset of studies (10%, maximum 200 references) retrieved by the search procedure will be processed independently by a second team member, who applies the same inclusion criteria. The final result will be analysed and documented using Kappa statistics ( Studies which were excluded by one, but included by the other reviewer will be documented and the reasons will be discussed in the review team. If the kappa-value is below 0.6 (substantial), a common strategy will be developed and inclusion criteria will be refined and tested to improve reviewer agreement and to minimize discrepancies in the screening process.

Study quality assessment

For each study fulfilling the inclusion criteria of the systematic review, the reliability will be evaluated by judging the risk to systematic bias as low, high, or uncertain. The results of this assessment will be provided.

Possible sources of bias will be addressed for each study including:

  • Selection bias (e.g. pre-treatment differences between study groups, differences in baseline characteristics):
    • ° Field management: High risk if fields or plots selected for the treatments have a different history of management/are owned by different growers, etc.

    • ° Dispersal of fields/plots: High risk if selected fields are not equally dispersed in a homogenous landscape (e.g. a group of Bt fields close together is compared to a spatially separated group of conventional fields close together; most Bt fields are near forests, while conventional fields are not or vice-versa; Bt fields are isolated from other (maize) crops, while conventional ones are not or vice-versa; distance between replicate experimental units is different for the different treatments (Bt crops; control crops); differences for treatments in soil types, neighbouring landscape, shading, altitude, slope, water conditions, etc.).

    • ° Replication: High risk if studies are not replicated or replicates of treatments are not randomized.

    • ° Insecticide applications: High risk if insecticides are applied shortly before the study so that diversity and abundance of non-targets is reduced in the studied fields and differences between treatments may be masked.

  • Performance bias (systematic differences between groups e.g. exposure to factors other than the intended intervention):
    • ° Plot size: High risk if plot size is small, in particular for highly mobile animals (e.g. flying species and large mobile species). As a consequence, treatments and controls cannot be separated and the study design does not allow the potential detection of differences between treatments.

    • ° Experimental design: Low risk for (randomized) block designs and replicated split field designs, high risk for unreplicated split-fields and for designs using separated fields (managed by different growers) for the different treatments. Generally, a low number of replicates indicates a high risk of performance bias.

    • ° Cultivars used: Low risk if corresponding near-isolines are used for Bt and conventional treatments, high risk if distantly related cultivars are used because then factors other than Bt may have an effect.

    • ° Expression of Bt protein: High risk if experimental (non-commercial) plants are used and the expression of Bt protein or their effect on the target pest(s) are not measured. Low risk if concentrations are measured or effect on target is demonstrated.

    • ° Application of pesticides: High risk if pesticides (other than insecticides) are potentially applied in one treatment group but not in others or if pesticide applications are not specified (e.g. herbicide or fungicide applications or insecticide applications not accounted for in the treatment comparisons).

  • Detection bias (the way outcomes are measured differs between treatment groups):
    • ° Sampling methods: High risk if different methods are used to measure animal populations in the different treatments (unlikely).

    • ° Type and application of methods: High risk if assessments (e.g. counts of individuals per plant) are done by eye and the persons doing the assessments knew how the plots/fields are treated. Lower risk for methods based on traps, such as sticky traps or pitfall traps, which can be evaluated more objectively. However, each trap type has its own limitations and the specimens collected by traps usually do not represent the actual numbers on plants. Furthermore, inadequate use of methods (e.g. inappropriate method for certain taxa, sampling off-peak, etc.) might bias the results.

    • ° Sampling occasions: High risk for low number of sampling dates: especially if population trajectories show a time shift between treatments.

  • Attrition bias (imbalance in sample sizes between study groups, missing data):
    • ° Sample size: High risk if the sample sizes differ between treatments (e.g. flooding of traps in some plots, etc.).

    • ° Missing data: High risk if substantially more missing data are present for one treatment than the other.

  • Reporting bias (e.g. preferential reporting of positive outcomes):
    • ° Study funding: Higher risk for studies conducted by institutions/organizations/companies with a commercial, political, or ideological interest in a certain outcome of their study (e.g. biotech company might not publish study showing adverse effects of their own product; NGOs with an anti-biotechnology agenda might not publish results showing no adverse effect of GM plant).

    • ° No-effect results may be less frequently published by researchers (file-drawer problem) and thus data may be biased towards studies demonstrating effects. In the area of GMO risk assessment, this is unlikely as studies can be published independent from the outcome. Wolfenbarger et al.[11] did not find an indication for publication bias in their meta-analyses.

For the assessment of study quality, the following characteristics of a study will be extracted and evaluated:

  •  Selection of field sites/plots (history of management before experiment; location in landscape) → selection bias

  •  Experimental design → selection bias and performance bias considering layout, randomization, replication, plot size

  •  Used cultivars, their relationship, and Bt protein concentrations → performance bias

  •  Pesticide applications in the treatment → performance bias

  •  Sampling method → detection bias

  •  Sample sizes → detection bias, attrition bias

  •  Study funding → reporting bias

Similar to the literature screening process, a random subsample of study quality assessments (10%, maximum 200) will be conducted by 2 members of the review team. The results will be compared with kappa statistics and discussed in the team to reach a consensus decision. Subgroup analyses will be performed that exclude studies with low quality to see if and how they influence the overall results. This will be documented to ensure transparency and reconstructability of the data analyses.

Data extraction strategy

Data extraction will be done in accordance with the previous meta-analysis by Naranjo (2009). All variables that will be extracted are listed in Table  2.
Table 2

List of variables to be extracted for the systematic review on non-target effects of Bt maize on non-target animals

Variable name



Closed terms


Unique identification number assigned to each publication




Author(s) of the listed publication




Year of publication of study




Citation, e.g. journal name, volume and page numbers




Title of the publication




Type(s) of institutions that the author(s) are affiliated with (e.g. academic, private sector, government, etc.)




Institution of corresponding author (or of first author if no corresponding author was listed)




Indicates whether study was published in a peer reviewed journal




Information on funding source of the study (government, private, mix)




Country where field study was performed




Indicates whether author provided additional details or data




Number of experiment within a study (e.g. different locations, years, etc.)




Figure number, table number or page number where means and variation were found




Indicates whether figures were scanned to obtain data values




Insecticidal Bt protein(s) engineered into the transgenic crop




Insect order targeted by the Bt protein(s)




Transgenic event of the crop tested




Transgenic hybrid or variety name




Non-transgenic hybrid or variety name




Non-target taxonomic class




Non-target taxonomic order




Non-target taxonomic family




Non-target taxonomic genus




Non-target taxonomic species



nontarget_finest grouping

Finest level of taxonomic resolution reported for the nontarget organism(s)




Functional group of nontarget organism(s)




Specifies if non-target is plant-, ground, or soil-dwelling




Codes flag for non-independence to be considered for analyses: TGLE = taxonomic group lumped elsewhere; EMUE = experimental means used elsewhere; CMUE = control means used elsewhere




Stage or sex of the non-target collected




Location of field(s) to the level of specificity provided by the author



Georeference longitude

GPS coordinates of the field (WGS84 decimal format)



Georeference latitude

GPS coordinates of the field (WGS84 decimal format)




Number of fields as described by the author




Cultivation practices used within the fields (notes on tillage, herbicides, fungicides, etc.)




Information on site particularities including pressure of target pests, weed infestation, disease infestation, soil characteristics, etc.




Size of replicate plots (in hectares)




Explanations for any calculations done to obtain plot size




Indicates whether the listed plot size is an average or an estimate




Indicates whether the authors indicated that they randomly assigned replicates to treatments




Date on which field plots were planted




Date on which first sample was taken




Date on which last sample was taken




More detailed description on study duration




Number of year known for the respective plots known to be grown with the Bt crop




Brand name of insecticides used




Active ingredient for insecticides used




Amount of active ingredient per spray




Indicates if insecticide was applied as spray, seed treatment, soil granules




Number of insecticide applications




Indicates if the number of insecticide applications is an average




Abbreviated description of sampling method




Detailed description of sampling method




Frequency of repeated samples per replicate field or plot




The number of times that each replicate field or plot was repeatedly sampled over the duration of the experiment




Number of subsamples per true replicate




Major category of response variable




Detailed description of response variable




True sample size for control treatment




True sample size for experimental treatment




Sample size for the control treatment as stated by the author




Sample size for the experimental treatment as stated by the author




Indicates whether values represent seasonal means across multiple sample days or means from peak days




Indicates whether we calculated the seasonal mean or peak days




Explains how we calculated the seasonal mean or peak days




Indicates whether the comparison is




1) untreated Bt vs. untreated control,


2) untreated Bt vs. insecticide treated control, or


3) treated Bt vs. treated control



Mean for the control treatment




Mean for the experimental treatment




Standard error for the control treatment




Standard error for the experimental treatment




Standard deviation for the control treatment




Standard deviation for the experimental treatment




Unit of measurement for the response variable




Statistical test used by author




Indicates whether a significant effect was detected by the author




Space for remarks for this record



Given is the variable name in the database, the definition of the variable, the type, and whether the variable content is restricted to closed (predefined) terms.

If original data are not available in a form directly usable for the database, estimations or calculations will be done. Those values will also be included in the database together with an explanation of how estimations and calculations were done. Examples are standard deviations calculated from standard errors or peak densities estimated from a seasonal trajectory. Information on the location of the data in the publication and how data were extracted or obtained (including contacts with authors) also will be given. If the presented data of a study are insufficient, authors will be contacted and asked to provide the required data.

The following rules will be applied to extracted data (according to [1012]):

  •  Data for comparisons of finest taxonomic resolution will be extracted

  •  For repeated measurements on the same plots, the seasonal means will be extracted. If they are not available, then values from the time point with the peak of the measured parameter for each treatment will be extracted (peak time may differ for control and Bt).

  •  All observations will be based on a single season. Thus differences in a parameter represent within-season differences and are not cumulative changes over years. Data from different years will be treated as different observations.

  •  If data on several life stages of the same species were collected in a study, data on the least mobile, but feeding stage will be used (usually larvae/juveniles).

  •  If data for one Bt line and multiple control lines are available, data of the near-isoline or closest related line to the Bt line will be used. If data for multiple Bt lines and one control line are available, the values of the control line will be used several times for comparisons with the Bt lines. If data for one Bt treatment and multiple control insecticide treatments are available, the values of the Bt line will be used several times. It will be noted in the database if one treatment is used for different comparisons. The following comparisons will be made: Bt untreated – Conventional untreated; Bt untreated – Conventional insecticide treated; Bt insecticide treated – Conventional insecticide treated.

In a first step, the data collected by Naranjo [12] will be checked to make sure the data were all recorded accurately. A random sample (25%) of newly entered data will be checked by a second member of the review team.

Data analysis

All identified datasets will be presented by means of a narrative summary of the extracted outcomes. All studies will be coded with keywords for certain categories. Tables and figures illustrating all existing evidence will be produced. Examples of how the data will be grouped and presented are: number of years covered, study year, first year and long-term cultivation with Bt-maize, crop and field margin, continent of study, country of study, spatial scale of experiment, experimental design, taxonomic group, functional group, parameter measured, Cry-proteins studied, and studies comparing Bt maize to insecticide treated or untreated conventional maize.

Statistical meta-analyses will be conducted for frequently studied taxonomic groups and measured parameters. Based on the knowledge gained from previous meta-analyses, a sufficient number of datasets is available for arthropods and the parameter abundance. Additional groups and/or parameters will be analysed if the amount of available data allows quantitative assessments (minimum of 5 observations per taxonomic group).

For meta-analyses, the following inclusion criteria will be applied:

  •  Clearly defined parameter (e.g. abundance of non-target animals) was measured in the field.

  •  Clearly defined taxonomic group or ecological function was measured.

  •  Means over plots/fields are available and a measure of variation (SE, SD) as well as sample size (N) is given for each treatment. Sample size ≥ 2 is required.

  •  Untreated Bt maize is compared to untreated or insecticide-treated conventional maize or insecticide-treated Bt maize is compared to treated conventional maize. Insecticide treatments include sprays, soil granules or seed coating, depending on the target pest(s).

  •  Original data are available from the publication (tables, figures) or from the authors upon request.

  •  Data are presented on an annual basis (not pooled for several years).

Measures of treatment effect

The response variable will be abundance of non-target animals [12]. If enough data are available, other response variables might be used (e.g. diversity indices, measures of ecological function like predation or parasitism). We will report Hedges’ d, a weighted mean effect size estimator that is calculated as the difference between an experimental (Bt) and control (non-Bt) mean response divided by a pooled standard deviation and multiplied by a small sample size bias correction term.

Bt and control maize fields will be compared for different ecological functional groups, main taxonomic groups, geographical regions (continents), different Bt proteins (Lepidoptera and Coleoptera specific), commercialized and non-commercialized events.

Separate analyses will be performed for insecticide treated or untreated Bt and conventional maize.

Unit of analysis issues

We will assess if the unit of intervention (Bt maize cultivation, insecticide treatment) is the same as the unit of analysis (plots or fields analysed in a study). For field experiments with Bt maize, clustering issues in the experimental design are not likely to occur.

A data issue that is likely to occur, however, is the multiple use of the same dataset for different comparisons. For data extraction, we defined the rule that if multiple Bt lines are compared to one control line, the values of the control line are used several times and if one Bt treatment and multiple control insecticide treatments are present, the values of the Bt line are used several times. Consequently, for global analyses (comprising all Bt proteins and/or all insecticides) the same datasets might be present in an analysis several times. Furthermore, data on higher taxonomic levels may include data on individual genera or species and might result in a multiple use of the same data. Those datasets will be flagged (see Table  2, variable replicate data issues) in the database during the data extraction process. The proportion of datasets reused several times in one analysis will be given and discussed. Analyses with and without the multiple use of datasets might be conducted within the sensitivity analyses described below.

Dealing with missing data

Studies in which relevant data were recorded but not reported in the publication will be identified and the authors will be contacted and asked if they were able to provide the missing information. If the relevant data (as specified in the inclusion criteria) cannot be obtained or calculated from the given data, the dataset will not be included in the meta-analyses. Any estimates and calculations will be documented in the database.

Quantitative synthesis

The results of the meta-analyses will be presented with the help of figures and tables. The effect size of animal abundance for comparisons of Bt crops with treated or untreated control crops, including 95% confidence intervals, will be illustrated in bar charts. Effect sizes significantly different from 0 as well as the total number of observations will be indicated. Effect sizes will be calculated such that negative effect sizes are associated with lower abundance (or another response parameter) on Bt crops compared with non-Bt controls [12].

A narrative discussion of the findings will be provided. The outcome of the study quality assessment will be discussed.

Assessment of heterogeneity

We will test for heterogeneity across studies using adequate statistical approaches. The influence of variation caused by studying different taxa, environmental conditions (countries, regions, continents), crop managements (tillage, irrigation, pesticide application, rotation), spatial scales, and experimental designs will be explored.

Investigation of heterogeneity

The influence of different parameters causing heterogeneity will be investigated. For parameters causing high heterogeneity, separated meta-analysis of subgroups may be conducted to isolate and help identify the causes of the heterogeneity.

Sensitivity analysis

Sensitivity analyses will be conducted to explore the

  •  Influence of individual studies: The outcome of the meta-analyses might be influenced by
    • ° studies that provide many datasets (many years, many comparisons, many collected taxa)

    • ° studies with high replication (N), which are weighed higher

  •  Influence of funding sources or affiliation of investigators on the overall results

  •  Influence of peer-reviewed vs. non peer-reviewed studies

  •  Influence of study quality (high vs. low risk studies)

  •  Influence of studies where peak-days were measured in comparison to studies where seasonal means were measured

  •  Influence of plot size

  •  Influence of data on species, genus, family, or order level

  •  Influence of the number of dates on which samples were collected

  •  Influence of different transformation events (discussion of differences in Bt protein concentration in different plant parts)

Assessment of publication bias

Effect sizes will be compared for publications with different funding types (see sensitivity analyses described above). Systematic differences would indicate a publication bias depending on funding source of a study.

In addition, Begg’s funnel plots might indicate if studies with low precision (high variation) diverge from the pooled mean to a greater extent than studies with high precision. Missing data around effect size 0 might indicate that studies showing any kind of effect (positive or negative) are more likely to get published than no effect studies (file drawer problem).



Bacillus thuringiensis


C entral A ccess D atabase for I mpact A nalyses of Crop Genetic Improvement Technologies (Data-portal developed within GRACE,


European Union


European Food Safety Authority


Genetically modified


Genetically modified organism


EU project: G MO R isk A ssessment and C ommunication of E vidence,


Research Information System Format (standardized data format for bibliographic data)


Standard deviation


Standard error


Web of Science (Citation index databases provided by Thomson Reuters).



The project has been funded within the GRACE consortium, a EU-FP7 programme project, Grant Agreement KBBE-2011-6-311957. We are grateful to all collaborators within GRACE who provided valuable comments on an earlier version of this review protocol.

Authors’ Affiliations

Agroscope, Institute for Sustainability Sciences ISS
USDA-ARS, Arid-Land Agricultural Research Center
Julius Kühn-Institut, Institute for Biosafety in Plant Biotechnology


  1. James C: Global status of commercialized biotech/GM crops: 2012. In ISAAA Brief 44. Ithaca: International Service for the Acquisition of Agri-Biotech Applications; 2012.Google Scholar
  2. Ramankutty N, Evan AT, Monfreda C, Foley JA: Farming the planet: 1. Geographic distribution of global agricultural lands in the year 2000. Glob Biogeochem Cycles 2008, 22: GB1003.View ArticleGoogle Scholar
  3. Meissle M, Romeis J, Bigler F: Bt maize and integrated pest management – a European perspective. Pest Manag Sci 2011, 67: 1049–1058.View ArticleGoogle Scholar
  4. Devos Y, Aguilera J, Diveki Z, Gomes A, Liu Y, Paoletti C, du Jardin P, Herman L, Perry JN, Waigmann E: EFSA’s scientific activities and achievements on the risk assessment of genetically modified organisms (GMOs) during its first decade of existence: looking back and ahead. Transgenic Res 2014, 23: 1–25. 10.1007/s11248-013-9741-4View ArticleGoogle Scholar
  5. Nienstedt KM, Brock TCM, van Wensem J, Montforts M, Hart A, Aagaard A, Alix A, Boesten J, Bopp SK, Brown C, Capri E, Forbes V, Köpp H, Liess M, Luttik R, Maltby L, Sousa JP, Streissl F, Hardy AR: Development of a framework based on an ecosystem services approach for deriving specific protection goals for environmental risk assessment of pesticides. Sci Total Environ 2012, 415: 31–38.View ArticleGoogle Scholar
  6. Sanvido O, Romeis J, Gathmann A, Gielkens M, Raybould A, Bigler F: Evaluating environmental risks of genetically modified crops: ecological harm criteria for regulatory decision-making. Environ Sci Pol 2012, 15: 82–91. 10.1016/j.envsci.2011.08.006View ArticleGoogle Scholar
  7. Garcia-Alonso M, Raybould A: Protection goals in environmental risk assessment: a practical approach. Transgenic Res 2014. published online doi:10.1007/s11248–013–9760–1Google Scholar
  8. European Food Safety Authority (EFSA): Guidance on the environmental risk assessment of genetically modified plants. EFSA Panel on Genetically Modified Organisms (GMO). EFSA J 2010, 8: 1879. [doi:10.2903/j.efsa.2010.1879]Google Scholar
  9. Romeis J, Meissle M, Sanvido O: Ecological Impact of genetically modified organisms. IOBC/WPRS Bull 2008, 33: 158.Google Scholar
  10. Marvier M, McCreedy C, Regetz J, Kareiva P: A meta-analysis of effects of Bt cotton and maize on nontarget invertebrates. Science 2007, 316: 1475–1477. 10.1126/science.1139208View ArticleGoogle Scholar
  11. Wolfenbarger LL, Naranjo SE, Lundgren JG, Bitzer RJ, Watrud LS: Bt crop effects on functional guilds of non-target arthropods: A meta-analysis. PLoS One 2008, 3: e2118. 10.1371/journal.pone.0002118View ArticleGoogle Scholar
  12. Naranjo SE: Impacts of Bt crops on non-target invertebrates and insecticide use patterns. CAB Rev Pers Agric Vet Sci Nutr Nat Resour 2009, 4: 11.Google Scholar
  13. Duan JJ, Lundgren JG, Naranjo S, Marvier M: Extrapolating non-target risk of Bt crops from laboratory to field. Biol Lett 2010, 6: 74–77. 10.1098/rsbl.2009.0612View ArticleGoogle Scholar
  14. Kohl C, Craig W, Frampton G, Garcia-Yi J, van Herck K, Kleter GA, Krogh PH, Meissle M, Romeis J, Spök A, Sweet J, Wilhelm R, Schiemann J: Developing a good practice for the review of evidence relevant to GMO risk assessment. IOBC/WPRS Bull 2013, 97: 49–56.Google Scholar
  15. Smets G, Spök A, GRACE team members: Assessing the evidence of health, environmental and socio-economic Impacts of GMOs - GRACE stakeholder consultation on good review practice in GMO impact assessment. Part 1: overall process and review questions. Available online: [Accessed 11 April 2014]


© Meissle et al.; licensee BioMed Central Ltd. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.