Skip to main content


We're creating a new version of this page. See preview

  • Systematic Review Protocol
  • Open Access

Are ladybird beetles (Coleoptera: Coccinellidae) affected by Bt proteins expressed in genetically modified insect-resistant crops? A systematic review protocol

Environmental Evidence20198:25

  • Received: 13 February 2019
  • Accepted: 1 June 2019
  • Published:



Ladybird beetles (Coleoptera: Coccinellidae) are abundant predatory species in many agroecosystems, are valued for their biological pest control functions, and have been recommended as test species for studies supporting the assessment of non-target effects of insect-resistant Bt crops. Although insecticidal Bt proteins are known to be highly specific against target pests, some recent laboratory studies reported putative toxic effects of Bt proteins on ladybird species. While such studies have been criticised because of methodological shortcomings or inconsistencies, they cast doubt on the insecticidal spectrum of activity of some Bt proteins. Performing a systematic review that synthesises all existing evidence on this controversial topic may help to resolve the remaining scientific uncertainties. The review question to be addressed by the systematic review is the following: Are ladybird beetles (Coleoptera: Coccinellidae) affected by Bt proteins expressed in genetically modified insect-resistant crops? The systematic review will focus on studies performed under controlled environmental conditions.


An extensive literature search will be conducted to identify the articles relevant to the review question. A wide range of electronic bibliographic databases, the internet search engine Google Scholar, and websites of specialized organizations will be searched. Citation searching, reference list-checking and searching of key journals will also be performed. The relevance of the identified articles will be assessed against a set of pre-defined eligibility criteria, following a two-step approach. In the first step, title and abstract (or summary) will be screened, whilst in the second step the full text of all remaining articles will be assessed by two members of the review team. All relevant studies will be subjected to an appraisal of external (generalisability) and internal (risk of bias) validity. Data from the selected studies will be extracted and synthesised in a narrative report. If a sufficient number of datasets generated with comparable experimental setup is available, statistical meta-analyses will be conducted on a range of comparisons and including sensitivity analyses.


  • Bt crops
  • Cry proteins
  • Genetic engineering
  • Insect-resistant plants
  • Systematic literature search
  • Meta-analysis
  • Non-target arthropods
  • Vip proteins


Genetically modified (GM) crops have been cultivated around the world since 1996. In 2017, the area devoted to GM crops reached 189.8 million ha [1]. More than half of this production involved crops that are engineered to express one or more insecticidal proteins (Cry or Vip proteins) from the bacterium Bacillus thuringiensis (Bt) for controlling lepidopteran and/or coleopteran pests. Before their commercial release, GM crops undergo an environmental risk assessment (ERA) to ensure that they do not cause unacceptable detrimental effects to the environment. One area of concern addressed in this risk assessment is the potential adverse impact on non-target arthropods (NTAs) and the ecosystem services they provide, including biological pest control by predators and parasitoids [2].

Ladybird beetles (Coleoptera: Coccinellidae) are important NTAs: they are abundant predatory species in many agroecosystems  [3, 4], are valued for their biological pest control functions [5] and have been recommended as surrogate species for studies to support the ERA of Bt crops  [68]. Most ladybirds feed preferably on aphids, which do not ingest significant amounts of Bt proteins  [9]. However, they also consume other available prey and, occasionally, pollen when prey is scarce  [5], and therefore they can be exposed to Bt proteins under field conditions  [7, 10]. Some species, such as Stethorus punctillum, are specialist predators of spider mites, which are known to contain high amounts of Bt protein when feeding on Bt plants, and are thus highly exposed to Bt proteins [11, 12]. Figure 1 describes a conceptual model on how insecticidal Bt proteins could result in harm to ladybird beetles when feeding directly on plant parts or indirectly when preying on herbivores that have ingested the Bt protein.
Fig. 1
Fig. 1

Interactions of Bt proteins produced by genetically modified (GM) plants with (non-target) ladybird beetles including the potential pathway to environmental harm, ecosystem services and environmental protection goals. The focus of the systematic review will be on the hazard of Bt proteins on ladybird beetles

Insecticidal Bt proteins expressed in field-grown GM crops are known to be highly specific against target pests, and the scientific evidence accumulated with the assessment of these plants over the last three decades has demonstrated their environmental safety towards NTAs  [13]. However, some recent laboratory studies reported putative toxic effects of Bt proteins on some ladybird species  [1416]. Although such studies have been criticised because of methodological shortcomings or inconsistencies  [1720], they cast doubt on the insecticidal spectrum of activity of some Bt proteins and pointed to uncertainties that may be resolved by synthesising all existing evidence on this controversial topic.

Systematic reviews are evidence synthesis approaches that can be used to support the risk assessment of GM crops  [21], especially when the available evidence shows different, or even, contradicting results. A common approach for synthesising and analysing the data collected from multiple individual studies included in a systematic review is to conduct a meta-analysis. Meta-analyses combine data from multiple studies, increasing the statistical power and overcoming the low level of replication of individual studies  [2224]. Although several meta-analyses on the effects of GM insect-resistant crops on different groups of NTAs have been published to date  [2532], none of them has focused particularly on ladybird beetles.

We intend to perform a systematic review on the effects of Bt proteins expressed in GM insect-resistant crops on ladybird beetles. Since the studies reporting putative toxic effects of Bt proteins on some ladybird species were conducted in the laboratory, the systematic review will focus on studies performed under controlled environmental conditions. The methodological quality of each relevant study will be assessed. If the number of datasets generated with comparable experimental setup is sufficient, statistical meta-analyses will be conducted.

Objective of the review

The objective of the review is to systematically search, critically appraise and synthesise all scientific literature on the effects of Bt proteins, i.e. Cry and Vip proteins, expressed in GM insect-resistant crops on ladybirds.

Review question

We plan to answer the following review question: Are ladybird beetles (Coleoptera: Coccinellidae) affected by Bt proteins expressed in GM insect-resistant crops?

Components of the review question

The review question has a PECO structure with the key elements consisting of:
  • Population All species belonging to the coleopteran family Coccinellidae, commonly known as ladybirds, ladybird beetles, or ladybugs;

  • Exposure Cry and Vip proteins (Bt proteins) expressed in GM insect-resistant crops for which there has been a regulatory approval for cultivation. This includes the Bt proteins as expressed in the GM plant and microbially-produced and purified Bt proteins (hereafter referred as to purified proteins);

  • Comparator Closely-related (non-Bt) plants for those studies using Bt plant material/tissue (e.g. leaves, pollen), or a negative control for those studies using purified Bt proteins (e.g. untreated artificial diet);

  • Outcome Effects of the Bt protein on life-history traits of the ladybird beetle (e.g. development, survival, reproduction) measured under controlled environmental conditions (i.e. laboratory and glasshouse studies).


Searching for articles

Search terms

The search will be structured to reflect the population and exposure elements of the review question because both elements can be clearly defined and translated into broad search terms. On the contrary, the comparator and outcome components of the review question will remain open and not be present in the search string to avoid narrowing the search too much and thus missing relevant articles.

To enhance the sensitivity of the search, a wide range of search terms will be used (Table 1). Search terms will cover possible synonyms, related terms, acronyms, spelling variants, lay and scientific terminology and translation issues, and specific search functions, such as truncation, wildcards, proximity operators and quotation marks for multi-word terms. The Thesaurus search tools in Biosis Citation Index (Clarivate Analytics, Philadelphia, USA), CAB Abstracts (CABI, Wallingford, UK) and Zoological Record (Clarivate Analytics) were used to ensure that terms from the controlled vocabulary are included (e.g. coccinellidae, ladybirds, ladybugs). Only English search terms will be used.
Table 1

Search terms corresponding to the population and exposure elements of the review question

Key elements—concept

Search terms


coccinellid*; ladybird*; ladybug*; ladybeetle*; “lady bird*”; “lady bug*”; “lady beetle*”; Adalia; Cheilomenes; Coccinella; Coleomegilla; Cryptolaemus, Harmonia; Hippodamia; Propylea; Psyllobora; Stethorus; nontarget; non-target; “non target”; NTO; NTOs; NTA; NTAs

Exposure—Bt protein

cry1*; “cry 1*”; cry2*; “cry 2*”; cry3*; “cry 3*”; vip3*; “vip 3*”; “Bt protein*”; “Bt toxin*”; “Cry* protein*”; “Cry* toxin*”; “Vip* protein*”; “Vip* toxin*”; “insecticidal protein*”; “insecticidal toxin*”; “insecticidal compound*”; “insecticidal substance*”; “insecticidal activ*”; “pesticidal protein*”; “pesticidal toxin*”; “pesticidal compound*”; “pesticidal activ*”

Exposure—intended trait

Bacillus thuringiensis”; “B. thuringiensis

Insect; insects; pest; pests; Lepidoptera*; Coleoptera*

Resistan*; protect*; toleran*

Exposure—plant species

crop*; plant*; cotton; Gossypium; cowpea; “Vigna unguiculata”; eggplant; aubergine; brinjal; “Solanum melongena”; maize; corn; “Zea mays”; potato; “Solanum tuberosum”; rice; “Oryza sativa”; soybean; soja; soya; “Glycine max”; sugarcane; “sugar cane”; Saccharum; tomato; “Solanum lycopersicum

Exposure—intended trait × plant species

Bt crop*”; “Bt plant*”; “Bt cotton”; “Bt cowpea”; “Bt eggplant”; “Bt aubergine”; “Bt brinjal”; “Bt maize”; “Bt corn”; “Bt potato”; “Bt rice”; “Bt soybean”; “Bt soja”; “Bt soya”; “Bt sugarcane”; “Bt sugar cane”; “Bt tomato”

Exposure—GMO general

GM; GE; transgen*; “genetic* modif*”; “genetic* transform*”; “genetic* manipulat*”; “genetic* improve*”; “genetic* engineer*”; “living modif*”

The symbol * denotes truncation; quotation marks (“ ”) will be used for multi-word terms

Search strings

The search strings will be composed of two parts, corresponding to the population and exposure elements of the review question, combined with the Boolean operator AND. Within each set, search terms will be linked with the Boolean operator OR (Tables 2, 3). This string will retrieve articles that contain at least one search term from each element.
Table 2

Search strings to be used in the electronic bibliographic databases hosted in the Web of Science platform



Search string

Key element—concept



TS = (coccinellid* OR ladybird* OR ladybug* OR ladybeetle* OR “lady bird*” OR “lady bug*” OR “lady beetle*” OR adalia OR cheilomenes OR coccinella OR coleomegilla OR crytolaemus OR harmonia OR hippodamia OR propylea OR psyllobora OR stethorus OR nontarget OR non-target OR “non target” OR NTO OR NTOs OR NTA OR NTAs)




TS = (cry1* OR “cry 1*” OR cry2* OR “cry 2*” OR cry3* OR “cry 3*” OR vip3* OR “vip 3*” OR “bt protein*” OR “bt toxin*” OR “cry* protein*” OR “cry* toxin*” OR “vip* protein*” OR “vip* toxin*” OR ((insecticidal OR pesticidal) NEAR/3 (protein* OR toxin* OR compound* OR substance* OR active*)))

Exposure—Bt protein



TS = (“bacillus thuringiensis” OR “b thuringiensis” OR (insect OR insects OR pest OR pests OR lepidoptera* OR coleoptera*) NEAR (resistan* OR protect* OR toleran*))

Exposure—intended trait



TS = (“bt crop*” OR “bt plant*” OR “bt cotton” OR “bt cowpea” OR “bt eggplant” OR “bt aubergine” OR “bt brinjal” OR “bt maize” OR “bt corn” OR “bt potato” OR “bt rice” OR “bt soybean” OR “bt soja” OR “bt soya” OR “bt sugarcane” OR “bt sugar cane” OR “bt tomato”)

Exposure—intended trait × plant species



TS = ((GM OR GE OR transgen* OR “genetic* modif*” OR “genetic* transform*” OR “genetic* manipulat*” OR “genetic* improve*” OR “genetic* engineer*” OR “living modif*”) NEAR/3 (crop* OR plant* OR cotton OR gossypium OR cowpea ORvigna unguiculata” OR eggplant OR aubergine OR brinjal OR “solanum melongena” OR maize OR corn OR “zea mays” OR potato OR “solanum tuberosum” OR rice OR “oryza sativa” OR soybean OR soja OR soya OR “glycine max” OR sugarcane OR “sugar cane” OR saccharum OR tomato OR “solanum lycopersicum”))

Exposure—GMO general NEAR plant species



#2 OR #3 OR #4 OR #5

Bt protein OR intended trait OR Intended trait × plant species OR (GMO general NEAR plant species)



#1 AND #6


The symbol * denotes truncation; NEAR/3 denotes words within three words of each other; quotation marks (“ ”) will be used for multi-word terms

Terms in italics were added after the revision of the manuscript and were thus not part of the scoping exercise

Table 3

Search strings to be used in Agricola, AGRIS, Open Access Theses and Dissertations and Google Scholar


Search string


(coccinellid? OR ladybird? OR ladybug? OR ladybeetle? OR “lady bird?” OR “lady bug?” OR “lady beetle?” OR nontarget OR “non target” OR NTO OR NTOs OR NTA OR NTAs) AND (“bacillus thuringiensis” OR Bt OR Cry? OR Vip? OR GM? OR GE OR “genetically modified” OR “genetically engineered” OR transgenic)


(1) ladybird* bt; (2) coccinellid* bt; (3) ladybird* transgenic; (2) coccinellid* transgenic

Open Access Theses and Dissertations

(coccinellid* OR ladybird* OR ladybug* OR ladybeetle* OR “lady bird*” OR “lady bug*” OR “lady beetle*” OR nontarget OR “non target” OR NTO OR NTOs OR NTA OR NTAs) AND (“bacillus thuringiensis” OR Bt OR Cry* OR Vip* OR GM* OR GE OR “genetically modified” OR “genetically engineered” OR transgenic)

Google Scholara

(1) ladybird bt; (2) ladybug bt; (3) ladybeetle bt (4) coccinellid bt

The symbols “*” and “?” denote truncation; quotation marks (“ ”) will be used for multi-word terms

aSearches will be performed using the open source software Publish or Perish. Since nested Boolean operators do not work in Publish or Perish, four searches will be performed using the all the words command. The first 200 results from each search will be considered

The search strategy will be adapted to the different information sources, i.e. Web of Science (WoS)  [33], Agricola  [34], Open Access Theses and Dissertations  [35], and Google Scholar ( The search strings that will be used in the electronic bibliographic databases hosted in Web of Science and Agricola, AGRIS, Open Access Theses and Dissertations and Google Scholar are presented in Tables 2 and 3, respectively. Searches in Google Scholar will be performed using the open source software Publish or Perish ( We selected Publish or Perish because it eases the process to export the results retrieved in Google Scholar and import them into EndNote. Since nested Boolean operators do not work in Publish or Perish, several searches will be performed using the all the words command. The first 200 results from each search will be considered, as suggested by Haddaway et al.  [36].


The searches are expected to retrieve articles written in English and articles written in other languages with at least a title, abstract/summary or keywords in English. Identified articles written in languages other than English will be translated as needed for further assessment of relevance and validity.

Time period

The search will be limited to those articles published from 1990 onwards. Since the commercial cultivation of GM crops started in 1996, the likelihood that relevant articles were published before 1990 is considered very low.

Estimating the comprehensiveness of the search

We performed a scoping exercise on 29 September 2017 with the aim of estimating the comprehensiveness of the search and validating the selected search string with the electronic bibliographic databases hosted in the WoS platform. For this purpose, 18 already-known relevant articles were selected: twelve articles that were listed in two reviews [20, 37] and six articles (published between 2014 and 2017) that had been previously identified by citation alert (see Additional file 1).

Individual searches were performed in five of the electronic databases that will be searched in the systematic review (i.e. WoS Core Collection, CAB Abstracts, Chinese Science Citation Database, BIOSIS Citation Index and Current Contents Connect), using the search string provided in Table 2. All retrieved lists of records were checked for the presence of the 18 relevant articles. The results of the scoping exercise are provided in the Additional file 2. WoS Core Collection and CAB Abstracts delivered the best results and were able to retrieve 16 articles each. All databases combined returned all 18 relevant articles. Therefore, the selected search string was considered valid. None of the 18 relevant articles were retrieved by the Chinese Science Citation Database. This was expected since none of them were published in journals indexed in the database. However, the Chinese Science Citation Database will still be searched for the systematic review since it might contain some other relevant articles which are not indexed in the other databases.

Electronic bibliographic databases

A broad range of relevant multi-disciplinary and subject-specific electronic bibliographic databases will be searched:
  • Web of Science (WoS) Core Collection (Clarivate Analytics). Contains most peer-reviewed scientific articles in English language. The following databases will be included: Science Citation Index Expanded, Conference Proceedings Citation Index—Science, Book Citation Index—Science, and Emerging Sources Citation Index;

  • BIOSIS Citation Index (Clarivate Analytics). Comprehensive reference database for life sciences;

  • CABI: CAB Abstracts (CABI). Comprehensive database that also includes more local and non-English articles;

  • Chinese Science Citation Database (Chinese Academy of Sciences). It provides scientific articles published in Chinese language (title and abstract in English);

  • Current Contents Connect (Clarivate Analytics). It contains peer-reviewed scientific articles in English language from different disciplines;

  • Data Citation Index (Clarivate Analytics). It contains data sets from a wide range of data repositories and connects them with the scientific literature to track data citation;

  • SciELO Citation Index (Sao Paulo Research Foundation for the cooperative publishing of open access journals on the internet). It provides access to scholarly literature in natural sciences, social sciences, arts and humanities published in leading open access journals from Latin American, Portugal, Spain and South Africa;

  • Zoological Records (Clarivate Analytics). It covers all aspects of animal research;

  • Agricola. It contains bibliographic records of materials acquired by the National Agricultural Library and cooperating institutions in agricultural and related sciences;

  • AGRIS (Food and Agriculture Organization of the United Nations). It facilitates access to articles, monographs, book chapters, and grey literature in the area of agriculture and related sciences;

  • Open Access Theses and Dissertations. Contains a collection of open access graduate theses and dissertations published around the world.

All databases, except Agricola, AGRIS, and Open Access Theses and Dissertations, are hosted in WoS.

Search engines

Google Scholar, which is a web search engine that indexes peer-reviewed journals and other scholarly literature, such as books, conference papers, theses and dissertations, and technical reports, will be searched.

Organisational websites

The websites of specialist organisations (regulatory agencies, industry organisations and civil society organisations) and web-based databases containing information on environmental effects of GMOs listed below will be searched.

Supplementary searches

Supplementary searches will be conducted to identify potential additional relevant articles not retrieved by other methods.

Citation searching

Citation searching will be conducted with those articles meeting the eligibility criteria and categorised as relevant after the screening process. All references citing each relevant article will be identified in WoS using the cited reference search tool (all eight WoS-hosted databases cited above will be considered) and screened.

Checking reference lists

The reference lists of those articles meeting the eligibility criteria and therefore categorised as relevant after the screening process will be checked.

Electronic-searching of key journals

The ten journals that yielded the highest number of records when performing the scoping exercise on 29 September 2017 will be checked electronically to identify very recent articles that have not yet been indexed in the electronic bibliographic databases. The journals are, in alphabetical order, Biological Control, Entomologia Experimentalis et Applicata, Environmental Entomology, IOBC/WPRS Bulletin, Journal of Applied Entomology, Journal of Economic Entomology, Pest Management Science, Plos One, Scientific Reports and Transgenic Research. The searches will be limited to articles published within the 12 months before the date of the search of the systematic review, and to articles published ahead of print.

Search update

A search update will be conducted if searches were performed more than 2 years prior to review completion. Search updates will be thoroughly documented and reported as described below.

Documenting and reporting the search process

The search will be fully documented and reported following the recommendations outlined in the European Food Safety Authority (EFSA) guidance on systematic review methodology and food/feed safety risk assessment  [23], the EFSA explanatory note on literature searching  [38] and the ROSES RepOrting standards for Systematic Evidence Syntheses [39]. The ROSES form is included as Additional file 3. The following details will be recorded and reported in the systematic review:
  • List of all information sources (e.g. electronic bibliographic databases, organisational websites, hand searched journals) searched;

  • The full search strategy for each information source (copied and pasted exactly as run) and details of how each information source was searched will be recorded and provided in the systematic review;

  • The date on which the search was conducted;

  • The date of the most recent update of the database that was searched;

  • The date span of the search;

  • Any limits applied to the search (e.g. article types, dates);

  • The journal name, the journal URL (internet address) or publisher; the dates, volumes and issues searched; the method of searching, e.g. scanning tables of contents for each issue, or using a search engine; the search terms used (if any) (for electronic-searching key journals);

  • The number of articles identified through each information source and the final number of articles remaining after removing duplicates (automatically and manually). This information will be part of the flow diagram that includes information on the screening and critical appraisal and synthesis steps [39];

  • Any deviations from the protocol in the search strategy, and their impact in the systematic review.

Article screening and study eligibility criteria

Screening process

The articles retrieved by the literature searches will be transferred to EndNoteX9 (Clarivate Analytics). Retrieved articles will be tagged with the database provider and stored in one single EndNote file. Then, the articles will be combined into one file that will be exported to the systematic review software Distiller SR (Evidence Partners, Ottawa, Canada). Duplicates will be eliminated automatically. Remaining duplicates, if any, will be removed manually.

The relevance of the resulting articles after deduplication will be assessed manually following a two-step approach and applying the eligibility criteria listed in Table 4. In the first step, title and abstract (or summary) will be screened, whilst in the second step, the full text of all remaining articles will be assessed. At this second step, each excluded article will be documented with the reason for exclusion. Any articles for which their relevance still remains unclear after assessing the full text will be discussed by the review team to reach a consensus decision.
Table 4

Eligibility criteria to assess the relevance of retrieved articles at title/abstract and full-text screening stage



Key elements of the review question


Studies include one or several ladybird species (Coleoptera: Coccinellidae)


Studies involve GM crops producing one or more Cry or Vip proteins from Bt and for which there has been a regulatory approval for cultivationa (i.e., cotton, cowpea, eggplant, maize, potato, rice, soybean, sugarcane, tomato), or purified Bt proteins. The Bt protein can be administered to the ladybird directly or indirectly (i.e. using a prey feeding on plant material or diet containing purified Bt protein)


Studies include a comparator to which the Bt plant or the Bt protein is compared (i.e. non-Bt plant or plant material/tissue in the case of plant studies, or a negative control in the case of studies with purified Bt proteins)


Studies report lethal (mortality or survival) and/or sub-lethal endpoints (e.g. growth, development, reproduction)

Additional concepts

 Study type

The article presents original/primary data (i.e. no reviews)

 Study design

Studies are performed under controlled environmental conditions [(extended) laboratory and glasshouse studies)]. Studies performed under field or semi-field conditions will not be eligible for further assessment


Searches will be conducted using English terms. Identified articles written in languages other than English will be translated and further assessed for their relevance and risk-of-bias

 Time period

Articles published from 1990 onwards

Bt: Bacillus thuringiensis

aGM Approval Database (

Screening consistency checking

Articles will be screened independently by at least two reviewers at the title, abstract, and full text level. To minimise any influence on eligibility decisions  [40], reviewers will not be allowed to assess the relevance of any article they have co-authored. Any articles considered relevant by one reviewer and non-relevant by the second one will be documented and discussed to reach a consensus decision. If no consensus is reached, the opinion of another member of the review team will be sought.

Eligibility criteria

From the articles retrieved by the different sources of scientific literature described above, only those articles fulfilling all the eligibility criteria listed in Table 4 will be considered relevant for the systematic review.

Pilot testing eligibility criteria

The validity of the eligibility criteria was pilot-tested on 6 June 2018, following the recommendations of Frampton et al.  [40]. For this purpose, a sample of 20 articles identified in the scoping exercise performed on 29 September 2017 was selected (see Additional file 4). This sample size is bigger than the size suggested by Higgins and Green  [41] (i.e. 10–12 articles). The sample included articles which were thought by one reviewer to be definitely relevant (10 articles), definitely irrelevant (seven articles), and doubtful (three articles). Articles were selected randomly but ensuring that all categories were represented. A second member of the review team assessed all articles independently. Both reviewers agreed on the relevance of all 20 articles assessed. Therefore, the defined eligibility criteria were considered valid.

Documenting and reporting article screening and eligibility criteria

The relevance assessment will be fully documented and reported following the recommendations in Frampton et al. [40], the ROSES RepOrting standards for Systematic Evidence Syntheses  [39], and the Collaboration for Environmental Evidence Guidelines and Standards for Evidence Synthesis in Environmental Management  [42].

The following details will be recorded and reported in the systematic review:
  • Summary of the finally-agreed eligibility criteria, including the instructions given to the reviewers;

  • A flow diagram showing the results of the screening: number of articles screened, number of articles excluded upon screening titles and abstracts (for both reviewers); number of articles excluded at full-text screening; number of articles used in the systematic review;

  • Number of reviewers involved; whether screening decisions were independent; expertise of the reviewers;

  • Results of the assessment of reviewer agreement, and how disagreements were resolved;

  • A list of articles that were excluded at full-text (with reasons for exclusion);

  • A list of articles which had unclear eligibility status after completion of full-text screening (with explanation why they could not be classified).

  • A list of articles that could not be obtained for full-text screening;

  • The final list of articles eligible for the systematic review;

  • Any deviations from the protocol in the eligibility criteria or the screening process, and their impact in the systematic review.

Study validity assessment

Study validity criteria and process

Relevant studies will be assessed for both external validity (the degree to which the studies are appropriate or applicable for answering the review question) and internal validity (risk of bias). The validity of each study fulfilling the eligibility criteria will be evaluated against a set of criteria. The study validity criteria are divided into three groups according to whether they pertain to: (1) test substance; (2) test organism, and (3) study design. A list of potential criteria that will be considered and the conditions that are needed to fulfil each one of them are provided in Table 5. Further details about the scientific rationale followed to underpin the selection of the proposed criteria can be retrieved from Romeis et al. [20, 43].
Table 5

Proposed criteria used to assess the validity of relevant studies



Conditions to be met

Test substance


The test substance is properly characterised and described. For studies with purified Bt proteins information on the source, purity, nominal concentration, batch/lot number, solvent used (vehicle), etc. should be provided. For studies with Bt plant material, information on the transformation event, the hybrid/variety, Bt protein expression levels, etc. should be given

Biological activitya

There is sufficient evidence that the test substance remains biologically active when provided to the test species. Biological activity can be demonstrated, for instance, by performing sensitive insect bioassays (e.g. Meissle and Romeis [44])


The test substance is biochemically and functionally equivalent to the Bt protein expressed in the GM plant. An overview of suitable methods to determine the equivalence between microbe- and plant-produced insecticidal proteins is given in Raybould et al. [45]


The test substance remains stable once incorporated into the artificial diet to ensure consistent exposure over the course of the study. Stability can be monitored by measuring Bt protein concentration throughout the duration of the study. Stability of the test substance can also be ensured by replacing the diet at regular intervals (e.g. Raybould and Vlachos [46])


The test substance is homogeneously distributed in the artificial diet to ensure that test organisms are not able to avoid the test substance altogether or are exposed to lower than expected Bt protein levels. Homogeneity of the diet can be determined, for instance, by analysing subsamples of the diet (e.g. Duan et al. [47])

Test organism

Life-stage testedc

Life-stages need to be selected that are most likely to be susceptible to the Bt protein protein and thus are most likely to detect an adverse effect. In general, neonates are more sensitive than later instars (Glare and O’Callaghan [48])

Study design

Negative controlc

The study includes a suitable negative control. This is essential to separate any background effects of the test system from effects due to the test substance (e.g. a diet identical to the test diet in all respects except the test substance in studies with purified Bt proteins, or a near-isogenic line in studies with Bt plant material)

Concentration/dose selectiona

The test organism is continuously exposed to the test substance throughout the duration of the study under worst-case conditions (i.e. ≥ 1 × EEC)

Test substance ingestiona

There is sufficient direct or indirect evidence that the test species has ingested the test substance. Ingestion can be confirmed directly by immune-assays such as ELISA, or indirectly, with the inclusion of a suitable positive control (e.g. Li and Romeis [12], Álvarez-Alfageme et al. [17]) or weighing the test organisms or food before and after feeding

Measurement endpointsc

Measurement endpoints are suitable to be evaluated in the laboratory/glasshouse and likely to indicate the possibility of adverse effects (e.g. mortality, fecundity, development duration, body mass, or the percentage of individuals that reach a certain life-stage)

Test durationc

The duration of the test considers the measurement endpoints, the biology and the life-stage tested of the test organism, and the characteristics and mode of action of the test substance

Experimental conditionsc

The experimental conditions (e.g. temperature, humidity, light:dark conditions) are appropriate for the test organisms and similar between the control and the treatment groups

Sample sizec

The sample size and the amount of missing data is similar between treatments

Statistical designc

The study employs a sufficient number of samples and replicates (e.g. based on power analyses), randomises treatments, ensures independence of observations and uses appropriate statistical methods

Bt: Bacillus thuringiensis; GM: genetically modified; EEC: expected environmental concentration; ELISA: enzyme-linked immunosorbent assay. Transformation event: insertion of DNA into the plant genome as a result of a single transformation process

aCriterion to assess external validity (generalisability)

bCriterion mostly relevant for studies with purified Bt proteins

cCriterion to assess internal validity (risk of bias)

For each study, all quality criteria will be assessed individually and categorised (e.g. fulfilled/low risk of bias; partially fulfilled/moderate risk of bias; not fulfilled/high risk of bias; not relevant; not assignable, in case that information is not provided or reported). An overall validity descriptor will be assigned to each study based on the results of the independent assessment of all criteria by two reviewers: (1) high validity/low risk of bias; (2) medium validity/medium risk of bias; (3) low validity/high risk of bias; or (4) not assignable.

In case that insufficient evidence is provided in the study to assess each of the validity criteria, corresponding authors will be contacted and asked for clarifications.

Details about the criteria assessed to determine the validity of the studies, the rationale followed to categorise each of criteria as well as the results of the assessment of each study will be provided in the systematic review. Information from the validity assessment of the eligible studies will be used in data synthesis; sensitivity analyses will be performed by comparing results with and without exclusion of studies with low and medium validity.

Consistency checking

A similar quality assurance system to that established for the article screening process will be put in place. All studies will be assessed independently by two members of the review team. Reviewers will not be allowed to assess the study validity of studies they have co-authored. Inconsistencies or uncertainties between reviewers will be discussed to reach a consensus decision. If no consensus is reached, the opinion of another member of the review team will be sought.

Documenting and reporting study quality

The rationale behind the decision for each validity criterium and the overall external and internal validity of each study will be documented in the systematic review, as well as any deviation from the protocol.

Data coding and extraction strategy

The variables that will be extracted from relevant studies and recorded in an MS-Excel spreadsheet are listed in Table 6. The extracted data will be available as an additional file of the systematic review.
Table 6

List of variables to be extracted from each relevant study


Variable name



Closed terms

Study details


Unique identifier assigned to each study




Type of study I



study type_iib

Type of study II




Author(s) of the study




Year of publication




Title of the study




Journal name, volume and page numbers




Type(s) of institutions that the author(s) are affiliated with (academic/private sector/government)




Indicates whether the study was published in a peer-reviewed journal




Information on funding source of the study (public/private/mixed)



Test substance

test_substance_ category

Category of the test substance tested (e.g. purified protein, pollen, leaf)




Bt protein tested




Insect order targeted by the Bt protein (Lepidoptera/Coleoptera)




Purity of the Bt protein tested in %



bt_protein_ concentration

Concentration of the Bt protein tested




Unit of measurement for the Bt protein concentration



bt_protein_ equivalenced

Indicates whether the microbially-produced/purified Bt protein tested is equivalent to the Bt protein produced by the GM plant (direct or indirect evidence)




More detailed description on Bt protein equivalence



bt_protein_ stabilityd

Indicates whether the microbially-produced/purified Bt protein tested was stable during the bioassay (direct or indirect evidence)



bt_protein_ stability_detailedd

More detailed description on Bt protein stability



bt_protein_ bioactivityd

Indicates whether the microbially-produced/purified Bt protein tested was biologically active (direct or indirect evidence)



bt_protein_ bioactivity_detailedd

More detailed description on Bt protein activity




Crop used in the study




GM event of the crop tested




Bt hybrid or variety name



non-bt_hybrid_or_ varc

Non-Bt hybrid or variety name




Information on the relatedness between the Bt and the non-Bt line used in the study



Test organism


Ladybird taxonomic genus




Ladybird taxonomic species




Life stage tested (larvae/adult)



Study design


Duration of the study in days



study_duration_ detailed

More detailed information on study duration




Information on the negative (non-Bt) control used in the study




Indicates whether a positive control was used




Prey taxonomic species



prey_species_ susceptibilitye

Indicates whether prey is susceptible to the test substance



prey_species_ susceptibility_ evidencee

Type of direct or indirect evidence provided to claim susceptibility of the prey species to the test substance



exposure_to_bt_ protein

Indicates whether exposure of test species to Bt protein was confirmed



exposure_to_bt_ protein_detailed

More detailed information on exposure to Bt protein



endpoint_ measured

Endpoint measured in the study



endpoint_ measured_ unit

Unit of measurement for the endpoint measured



endpoint_measured_ detailed

Detailed description of endpoint measured




Sample size for the control (non-Bt) treatment




Sample size for the experimental (Bt) treatment




Statistical test used by the author(s)





Mean for the control treatment




Mean for the experimental treatment




Standard error for the control treatment




Standard error for the experimental treatment




Standard deviation for the control treatment




Standard deviation for the experimental treatment




Figure, table or page number where means and variation were found




Indicates whether figures were scanned to obtain data values



significant_dif_ observed

Indicates whether a statistically significant effect was identified by the authors





Space for comments for this record



Bt: Bacillus thuringiensis; GM: genetically modified; na: not available

aLaboratory, extended laboratory, glasshouse

bType 1: direct and purified Bt protein; Type 2: direct and Bt plant; Type 3: indirect (prey-mediated) and purified Bt protein; Type 4: indirect (prey-mediated) and Bt plant

cOnly relevant for studies with Bt plant material

dOnly relevant for studies with purified Bt proteins

eOnly relevant for indirect (prey-mediated) studies

A random sample of entered data (at least 20% of the remaining studies after relevance assessment) will be checked by a second member of the review team.

The corresponding authors of those studies in which relevant data are not properly reported might be contacted; alternatively, data figures will be scanned using the open source software Plot Digitizer ( and means and measures of within treatment variance will be estimated.

Potential effect modifiers and reasons for heterogeneity

Several potential effect modifiers and sources of heterogeneity were identified:
  • Test substance
    • Protein type: Cry/Vip protein

    • Protein specificity: coleopteran-/lepidopteran-specific

    • Purified protein/plant-expressed protein

  • Test organism
    • Ladybird beetle species

    • Stage tested: larvae, pupae, adults

  • Study type/design
    • Study type: laboratory/glasshouse; direct feeding study/indirect (prey-mediated) feeding study; artificial diet/plant study

    • Measurement endpoint

    • Exposure duration

Those effect modifiers will be part of the data extracted from the studies and recorded in the MS Excel spreadsheet. Additional effect modifiers may be added during the course of the systematic review. The influence of the potential effect modifiers causing heterogeneity will be investigated.

Data synthesis and presentation

The data extracted from all identified studies will be synthesised in a narrative report and in different tables, including information on the ladybird species tested, type of study (e.g. direct/indirect feeding assay with plant material/purified protein in artificial diet), characterisation of the test substance, exposure to the test substance, endpoints analysed, negative and positive controls and results of the statistical analysis. The data extracted will describe the body of evidence and will also serve to identify potential knowledge/data gaps.

If a sufficient number of datasets generated with comparable experimental setup is available, statistical meta-analysis will be conducted using the effect size estimator Hedge’s d for continuous variables (e.g., developmental time, pupal/adult weight) and odds ratio and/or risk ratio for dichotomous variables (e.g., survival/mortality).

Separate analyses will be conducted for different study types, type of Bt protein, spectrum of activity of the Bt protein, measurement endpoint, etc.

Sensitivity analysis will be conducted to explore the influence of individual studies, funding sources or affiliation of investigators for the overall result and study validity (i.e., by comparing results with and without exclusion of studies with low and medium validity), etc.

Publication bias will be estimated; effect sizes will be compared for articles with different funding and author affiliation types (see sensitivity analyses described above). Systematic differences would indicate a publication bias depending on funding source of a study or authors affiliation.



Not applicable.


Any views expressed in this article are the views of the authors and do not necessarily represent the views of any organization or institution, with which they are currently affiliated or employed.

Authors’ contributions

This manuscript was drafted by FAA. YD, IMG, YL, MM, and JR provided comments. All authors read and approved the final manuscript.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

European Food Safety Authority, Via Carlo Magno 1A, 43126 Parma, Italy
State Key Laboratory for Biology of Plant Diseases and Insect Pests, Institute of Plant Protection, Chinese Academy of Agricultural Sciences, Beijing, 100193, People’s Republic of China
Agroscope, Research Division Agroecology and Environment, Reckenholzstrasse 191, 8046 Zurich, Switzerland


  1. ISAAA. Global status of commercialized biotech/GM crops: 2017. ISAAA Brief 53. 2017.Google Scholar
  2. Devos Y, Romeis J, Luttik R, Maggiore A, Perry JN, Schoonjans R, Streissl F, Tarazona JV, Brock TCM. Optimising environmental risk assessments accounting for ecosystem services helps to translate broad policy protection goals into specific operational ones for environmental risk assessments. EMBO Rep. 2016;16:1060–3.View ArticleGoogle Scholar
  3. Meissle M, Álvarez-Alfageme F, Malone LA, Romeis J. Establishing a database of bio-ecological information on non-target arthropod species to support the environmental risk assessment of genetically modified crops in the EU. EFSA Support Publ. 2012;EN-334:1–170.Google Scholar
  4. Riedel J, Romeis J, Meissle M. Update and expansion of the database of bio-ecological information on non-target arthropod species established to support the environmental risk assessment of genetically modified crops in the EU. EFSA Support Publ. 2016;EN-956:1–109.Google Scholar
  5. Hodek I, van Emden HF, Honěk A, editors. Ecology and behaviour of the ladybird beetles (Coccinellidae). Chichester: Wiley; 2012.Google Scholar
  6. Carstens K, Cayabyab B, De Schrijver A, Gadaleta PG, Hellmich RL, Romeis J, Storer N, Valicente FH, Wach M. Surrogate species selection for assessing potential adverse environmental impacts of genetically engineered insect-resistant plants on non-target organisms. GM Crops Food. 2014;5:11–5.View ArticleGoogle Scholar
  7. Li Y, Zhang Q, Meissle M, Yang Y, Wang Y, Hua H, Peng Y, Romeis J. Bt rice in China—focusing the non-target risk assessment. Plant Biotechnol J. 2017;15:1340–5.View ArticleGoogle Scholar
  8. Romeis J, Meissle M, Álvarez-Alfageme F, Bigler F, Bohan D, Devos Y, Malone LA, Pons X, Rauschen S. Potential use of an arthropod database to support the nontarget risk assessment and monitoring of transgenic plants. Transgenic Res. 2014;23:995–1013.View ArticleGoogle Scholar
  9. Romeis J, Meissle M. Non-target risk assessment of Bt crops—cry protein uptake by aphids. J Appl Entomol. 2011;135:1–6.View ArticleGoogle Scholar
  10. Harwood JD, Samson RA, Obrycki JJ. Temporal detection of Cry1Ab endotoxins in coccinellid predators from fields of Bacillus thuringiensis corn. Bull Entomol Res. 2007;97:643–8.View ArticleGoogle Scholar
  11. Álvarez-Alfageme F, Ferry N, Castañera P, Ortego F, Gatehouse AMR. Prey-mediated effects of Bt maize on fitness and digestive physiology of the red spider mite predator Stethorus punctillum Weise (Coleoptera: Coccinellidae). Transgenic Res. 2008;17:943–54.View ArticleGoogle Scholar
  12. Li Y, Romeis J. Bt maize expressing Cry3Bb1 does not harm the spider mite, Tetranychus urticae, or its ladybird beetle predator, Stethorus punctillum. Biol Control. 2010;53:337–44.View ArticleGoogle Scholar
  13. National Academies of Sciences, Engineering, and Medicine. Genetically engineered crops: experiences and prospects. Washington, DC: The National Academies Press; 2016.Google Scholar
  14. Dhillon MK, Sharma HC. Effects of Bacillus thuringiensis δ-endotoxins Cry1Ab and Cry1Ac on the coccinellid beetle, Cheilomenes sexmaculatus (Coleoptera, Coccinellidae) under direct and indirect exposure conditions. Biocontrol Sci Technol. 2009;19:407–20.View ArticleGoogle Scholar
  15. Hilbeck A, McMillan JM, Meier M, Humbel A, Schläpfer-Miller J, Trtikova M. A controversy re-visited: is the coccinellid Adalia bipunctata adversely affected by Bt toxins? Environ Sci Eur. 2010;24:10.View ArticleGoogle Scholar
  16. Schmidt JEU, Braun CU, Whitehouse LP, Hilbeck A. Effects of activated Bt transgene products (Cry1Ab, Cry3Bb) on immature stages of the ladybird Adalia bipunctata in laboratory ecotoxicity testing. Arch Environ Contam Toxicol. 2009;56:221–8.View ArticleGoogle Scholar
  17. Álvarez-Alfageme F, Bigler F, Romeis J. Laboratory toxicity studies demonstrate no adverse effects of Cry1Ab and Cry3Bb1 to larvae of Adalia bipunctata (Coleoptera: Coccinellidae): the importance of study design. Transgenic Res. 2011;20:467–79.View ArticleGoogle Scholar
  18. Rauschen S. A case of “pseudo science”? A study claiming effects of the Cry1Ab protein on larvae of the two-spotted ladybird is reminiscent of the case of the green lacewing. Transgenic Res. 2010;19:13–6.View ArticleGoogle Scholar
  19. Ricroch A, Bergé JB, Kuntz M. Is the German suspension of MON810 maize cultivation scientifically justified? Transgenic Res. 2010;19:1–12.View ArticleGoogle Scholar
  20. Romeis J, McLean M, Shelton AM. When bad science makes good headlines: Bt maize and regulatory bans. Nat Biotechnol. 2013;31:386–7.View ArticleGoogle Scholar
  21. Kohl C, Frampton G, Sweet J, Spök A, Haddaway NR, Wilhelm R, Schiemann J. Can systematic reviews inform GMO risk assessment and risk management? Front Bioeng Biotechnol. 2015;3:113.View ArticleGoogle Scholar
  22. Arnqvist G, Wooster D. Meta-analysis: synthesizing research findings in ecology and evolution. Trends Ecol Evol. 1995;10:236–40.View ArticleGoogle Scholar
  23. EFSA (European Food Safety Authority). Application of systematic review methodology to food and feed safety assessments to support decision-making. EFSA J. 2010;8:1–90.Google Scholar
  24. Marvier M. Using meta-analysis to inform risk assessment and risk management. J Verbrauch Lebensm. 2011;6(Supplement 1):113–8.View ArticleGoogle Scholar
  25. Comas C, Lumbierres B, Pons X, Albajes R. No effects of Bacillus thuringiensis maize on nontarget organisms in the field in southern Europe: a meta-analysis of 26 arthropod taxa. Transgenic Res. 2014;23:135–43.View ArticleGoogle Scholar
  26. Dang C, Lu Z, Wang L, Chang X, Wang F, Yao F, Peng Y, Stanley D, Ye G. Does Bt rice pose risks to non-target arthropods? Results of a meta-analysis in China. Plant Biotechnol J. 2017;15:1047–53.View ArticleGoogle Scholar
  27. Duan JJ, Marvier M, Huesing J, Dively G, Huang ZY. A meta-analysis of effects of Bt crops on honey bees (Hymenoptera: Apidae). PLoS ONE. 2008;3(1):e1415.View ArticleGoogle Scholar
  28. Marvier M, McCreedy C, Regetz J, Kareiva P. A meta-analysis of effects of Bt cotton and maize on nontarget invertebrates. Science. 2007;316(5830):1475–7.View ArticleGoogle Scholar
  29. Naranjo SE. Impacts of Bt crops on non-target organisms and insecticide use patterns. In: CAB reviews. Perspectives in agriculture, veterinary science, nutrition and natural resources, vol. 4. 2009. p. 11.Google Scholar
  30. Pellegrino E, Bedin S, Nuti M, Ercoli L. Impact of genetically engineered maize on agronomic, environmental and toxicological traits: a meta-analysis of 21 years of field data. Sci Rep. 2018;8:3113.View ArticleGoogle Scholar
  31. Romeis J, Meissle M, Naranjo S, Li Y, Bigler F. The end of a myth—Bt (Cry1Ab) maize does not harm green lacewings. Front Plant Sci. 2014;5:391.View ArticleGoogle Scholar
  32. Wolfenbarger LL, Naranjo SE, Lundgren JG, Bitzer RJ, Watrud LS. Bt crop effects on functional guilds of non-target arthropods: a meta-analysis. PLoS ONE. 2008;3(5):e2118.View ArticleGoogle Scholar
  33. Web of Science. Clarivate Analytics, USA. Accessed 25 July.
  34. Agricola National Agricultural Library. Department of Agriculture, USA. Accessed 14 May 2019.
  35. Open Access Theses and Dissertations. Accessed 14 May 2019.
  36. Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS ONE. 2015;10(9):e0138237.View ArticleGoogle Scholar
  37. Li Y, Romeis J, Wu K-M, Peng Y-F. Tier-1 assays for assessing the toxicity of insecticidal proteins produced by genetically engineered plants to non-target arthropods. Insect Sci. 2013;21:125–34.View ArticleGoogle Scholar
  38. EFSA (European Food Safety Authority), Devos Y, Guajardo IM, Álvarez F, Glanville J. Explanatory note on literature searching conducted in the context of GMO applications for (renewed) market authorisation and annual post-market environmental monitoring reports on GMOs authorised in the EU market. EFSA Support Publ. 2019;EN-1614:1–62.Google Scholar
  39. Haddaway NR, Macura B, Whaley P, Pullin AS. ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ Evid. 2018;7:7.View ArticleGoogle Scholar
  40. Frampton GK, Livoreil B, Petrokofsky G. Eligibility screening in evidence synthesis of environmental management topics. Environ Evid. 2017;6:27.View ArticleGoogle Scholar
  41. Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Chichester: Wiley-Blackwell; 2011.Google Scholar
  42. Collaboration for Environmental Evidence. Guidelines and standards for evidence synthesis in environmental management. Version 5.0. In: Pullin AS, Frampton GK, Livoreil B, Petrokofsky G, editors. 2018. Accessed 14 May 2019.
  43. Romeis J, Hellmich RL, Candolfi MP, Carstens K, De Schrijver A, Gatehouse AMR, Herman RA, Huesing JE, McLean MA, Raybould A, Shelton AM, Waggoner A. Recommendations for the design of laboratory studies on non-target arthropods for risk assessment of genetically engineered plants. Transgenic Res. 2011;20:1–22.View ArticleGoogle Scholar
  44. Meissle M, Romeis J. Insecticidal activity of Cry3Bb1 expressed in Bt maize on larvae of the Colorado potato beetle, Leptinotarsa decemlineata (Coleoptera: Chrysomelidae). Entomol Exp Appl. 2009;131:308–19.View ArticleGoogle Scholar
  45. Raybould A, Kilby P, Graser G. Characterising microbial protein test substances and establishing their equivalence with plant-produced proteins for use in risk assessments of transgenic crops. Transgenic Res. 2013;22:445–60.View ArticleGoogle Scholar
  46. Raybould A, Vlachos D. Non-target organism effects tests on Vip3A and their application to the ecological risk assessment for cultivation of MIR162 maize. Transgenic Res. 2010;20:599–611.View ArticleGoogle Scholar
  47. Duan JJ, Teixeira D, Huesing JE, Jiang C. Assessing the risk to nontarget organisms from Bt corn resistant to corn rootworms (Coleoptera: Chrysomelidae): tier-I testing with Orius insidiosus (Heteroptera: Anthocoridae). Environ Entomol. 2008;37:838–44.View ArticleGoogle Scholar
  48. Glare TR, O’Callaghan M. Bacillus thuringiensis: biology, ecology and safety. Chichester: Wiley; 2000.Google Scholar


© The Author(s) 2019