Skip to content

Advertisement

  • Commentary
  • Open Access

Inclusive development and prioritization of review questions in a highly controversial field of regulatory science

Environmental EvidenceThe official journal of the Collaboration for Environmental Evidence20187:1

https://doi.org/10.1186/s13750-017-0113-z

Received: 28 February 2017

Accepted: 25 November 2017

Published: 12 January 2018

Abstract

How to best assess potential health, environmental and other impacts of genetically modified organisms (GMOs) and how to interpret the resulting evidence base have been long-standing controversial issues in the EU. As a response, transparency and inclusiveness became a major focus of regulatory science activities in the GMO impact area. Nevertheless, nearly three decades of controversies resulted in a heavily polarized policy environment, calling for further efforts. Against this backdrop the EU funded project GRACE explored the value of evidence synthesis approaches for GMO impact assessment and developed an evidence synthesis framework with a strong emphasis on openness, stakeholder engagement, transparency, and responsiveness to tackle regulatory science challenges. This framework was tested and implemented in the course of 14 systematic reviews or maps conducted on selected review questions spanning potential health, environmental, and socioeconomic impacts of GMOs. An inclusive development and prioritisation of review questions is of key importance in evidence synthesis as it helps to provide a better link between stakeholder demands and concerns and policy relevant outcomes. This paper, therefore, places a particular focus on the stakeholder involvement strategy developed and experiences gathered during this particular step in the course of the GRACE project. Based on this experience, possible lessons for future engagement exercises in highly controversial fields of regulatory science are discussed.

Keywords

Evidence synthesisSystematic reviewsTransparencyInclusivenessStakeholder involvementGMO impact assessmentGMO risk assessmentPrioritization of review questions

Background

Evaluating potential health, environmental and socioeconomic impacts of genetically modified (GM) crops has been a dynamic field of research since the early 1990s. Interpreting the resulting primary data has frequently been a subject of controversial debates augmented by the complexity and the diversity of test designs and the multitude of endpoints under investigation [1] as well as by contradicting results. Reviewing secondary data by academia and regulatory committees has frequently led to discussions about, among other, studies not (appropriately) considered, relative weight attached to data generated by different methods, bias, lack of transparency and to divergent conclusions [28].

Against this backdrop the EU funded research project GRACE (GMO Risk Assessment and Communication of Evidence) set out to explore the use of evidence synthesis approaches to assess and synthesize exiting evidence on potential health, environmental and socio-economic impacts of GM crops [9]. At the time of the planning and start of this project (2011–2012) systematic reviews and maps had sparingly been applied to GMO impact research.

GRACE had a twofold aim: (i) to explore the suitability of systematic reviews and maps in GMO impact research and assessment, and (ii) to assess the available evidence on selected research questions. The results were expected to support evidence-based policy making in GMO impact assessment.

By adapting existing guidance documents [1012] GRACE developed a general framework appropriate for the synthesis of GM crop impact data [13]. Based on this general framework 14 systematic reviews and maps were planned and started in parallel.

A particular emphasis was placed on an active stakeholder involvement going beyond what is generally done in the course of systematic reviews and maps. A multi-step stakeholder engagement approach was developed inspired by two main considerations: (i) the long-standing polarization of the GMO impact debate which is accompanied by a lack of trust among stakeholders and (ii) the concept of Responsible Research and Innovation (RRI) underlying the recent research programmes funded by the European Commission and many national agencies [1416]. The resulting engagement approach, aiming to strengthen the openness, inclusiveness, responsiveness, and transparency of systematic reviews, was implemented for all GRACE reviews and maps in a harmonised way by a team specifically set for the purpose, involving researchers with expertise in natural and social sciences.

According to participant surveys and authors’ experiences, stakeholder participation was perceived to be most productive and important in the development and selection of the review questions. As shown in this paper, stakeholders had a considerable impact on the definition and selection of research questions and, thereby, on the scope of synthesis results. These steps are, therefore, the focus of this paper.

Characteristics of the GRACE stakeholder engagement approach for planning the reviews and maps

An overview of the stakeholder engagement approach is provided in Fig. 1.
Figure 1
Fig. 1

The GRACE approach to stakeholder engagement in the planning stage of evidence synthesis. a: Two review protocols on macro level socio-economic impacts were not published as available journals specialised on publishing such protocols considered them to be out of scope

Openness

Participation in stakeholder workshops was open for all interested stakeholder representatives following a broad invitation circulated to some 500 stakeholder contacts encompassing a broad range of, among others, competent authorities, industry, civil society organisations, professional organisations and researchers. Subsequent steps mainly involved workshop participants.

Despite limited resources, considerable attempts were made to have the key stakeholder groups represented across all steps.

Inclusiveness

As a first step, the overall framework and preliminary review questions were discussed in a 2-day workshop. Stakeholders could provide written comments and questions on the workshop discussions but also on issues not discussed there. Written comments were open to a wide audience including, but not limited to, workshop participants.

The resulting lists of the candidate review questions were prioritized in a second step by stakeholders using an online questionnaire, helping thus the project team identify a final set of review questions.

In the third step review protocols were developed and subjected, as drafts, to stakeholder comments in the course of a written consultation. Following a revision of the protocols prompted by stakeholder comments, they were published or prepared for publication as peer-reviewed journal papers.

In the fourth step preliminary results of the individual reviews as well as draft general conclusions and recommendations were presented and discussed in the course of another 2 day stakeholder workshop and again subjected to written comments.

Responsiveness

Stakeholders participating in the workshops as well as those provided with consultation materials could provide comments to the review team, which were then collected, categorised and subjected to the individual teams overview through the project’ s internal clearing house mechanism. Review teams discussed all comments, revised the review questions and developed written point-by-point responses to all comments and questions.

Transparency

Workshop participants were provided with draft documents and presentations before the workshop meetings. Each step was documented in a detailed stakeholder consultation report which covered workshop inputs and discussions, survey results, written stakeholder questions and comments as well as review team responses. Workshop reports documenting discussions were circulated as drafts to all participants in order to ensure that discussions and viewpoints were accurately depicted. The workshops, the written stakeholder comments, and the review teams’ responses are documented in detail in a series of consultations reports available [1721] on the project website [9].

Reflexivity

To better understand how the GRACE approach was perceived by participants, their views were gathered via phone interviews as well as via online and paper questionnaires.

Status of GRACE systematic reviews and maps

The characteristics described above also accompanied the stakeholder consultations on the preliminary results of the individual reviews and maps (not shown as they are beyond the scope of this paper). Furthermore, GRACE developed general conclusions and recommendations on the use and value of evidence synthesis approaches in GMO impact assessment [2224] which were also included in the scope of the consultations. By the time of the submission of this paper, 7 of the 14 review/map protocols [2531] and the results of one systematic review [32] have been published as peer reviewed open-access papers and 10 review/map protocols and the published systematic review are available on the Open Access database CADIMA [33].

Development and prioritization of review questions

The main steps in the development and prioritization of review questions are depicted in Fig. 1—indicating also the methodology used. The manner in which this process influenced the evolution of review questions is indicated in Table 1 and illustrated in Fig. 2.
Table 1

Development and selection of review questions

GMO impact area

Development and selection of review questions

Preliminary review questions

Changes made to review questions

Candidate review questions

Selected review questions

Revised

Dropped

Added

Health impacts

11

4

6

4

9

4a

Environmental impacts

23

4

3

0

20

6b

Socio-economic impacts

33

15

2d

9

40

4c

Total

67

23

11

13

69

14

Numbers indicate how many review questions entered into the process, were modified and selected. The review questions selected for conducting systematic reviews and maps are detailed in Table 2

aAll four review questions were revisited based on stakeholder comments

bThe prioritization exercise initially produced 11 questions, of which 3 were kept unchanged, 2 were further modified, 6 were dropped and another one added

c27 review questions on micro level socio-economic impacts were combined into one broad systematic map; 4 review questions on macro level socio-economic impacts were selected based on 14 candidate questions—two of the four questions were combined into one review question

d2 questions were dropped as separate questions but included in the scope of other review questions

Figure 2
Fig. 2

Example how stakeholder priorities and comments influenced type, scope and wording of review questions (described in detail for all review questions in [1720])

(The bar chart was reproduced with permission from [19])

Proposing and discussing review questions

In the first step a total of 68 preliminary review questions covering the three fields of GMO impact research were subjected to a consultation with 41 stakeholders. During this step the focus was on understanding and discussing the general characteristics of systematic reviews and maps, the potential relevance of this method for GMO impact research and assessment, the conceptual frameworks for the review questions, and the preliminary review questions.

Based on the workshop discussions and more than 170 written stakeholder comments received after the workshop, the project team modified the review questions considerably in terms of subject, scope, wording and numbers (summarised in Table 1). Of a total of 68 review questions, 23 were revised, 11 dropped and 13 new questions added. For the health and environmental impacts work streams this resulted in a reduction of the number of review questions. For the socio-economic impacts work stream it led to a broadening of the scope and to adding more review questions.

The resulting 69 candidate review questions were then prioritized by stakeholders in the next step.

Developing criteria and a process for prioritizing review questions

The method and criteria used were tailored to accommodate (i) the specific challenges of dealing with a large number of review questions in three very different scientific and policy contexts in parallel, (ii) the fact that this prioritization was done by stakeholders in a very polarized policy environment, (iii) the constraints of a relatively inflexible research project with tight timelines and resource limitations, and (iv) the exploratory nature of the project in applying evidence synthesis to GMO impact research.

The criteria used were (i) importance for impact assessment, (ii) the existence of expert controversy, and (iii) degree of public awareness (inspired by O’Connner et al. [34] and Clavesi et al. [35]). These three criteria refer to three important dimensions, which also act as three different contexts in the GMO debate: scientific, regulatory, and public dimension. A scoring system (1–5) was used with the option for adding comments allowing for a more nuanced feed-back.

Prioritization of the candidate review questions

The stakeholders participating in the previous consultation step plus those who expressed a particular interest (n = 55) were asked to score each of the 69 candidate review questions in each of the three criteria by using an electronic questionnaire (LimeSurvey).

The respondent rate of the questionnaire was 34.5% (health and environmental impacts) and 29.1% (socio-economic impacts) respectively. Aside from scores, stakeholders provided 74 additional comments. The resulting data set was analysed using SPSS and Excel. Mean scores for each criterion as well as aggregated scores (sum of mean scores of each criterion—assuming an equal relative weight of each criterion) were calculate for each candidate review question and where discussed and considered along with additional stakeholder comments by the three project teams (health, environmental and socioeconomic). An example of how stakeholder scores were reported to the review teams is provided in Fig. 3.
Figure 3
Fig. 3

Example of scores resulting from stakeholder prioritization. a Review questions 1–4 on Bt crops and Cry toxin. Means of scores for each criterion (importance, expert disagreement, and public awareness) on a scale of 1 (lowest) to 5 (highest). Criteria: importance: the review question is of high importance for the impact assessment of GMOs; expert disagreement: There is expert disagreement on the review question; Public awareness: The review question is the subject of high public awareness. b Review questions 1–4 on Bt crops and Cry toxin 1–4: aggregated mean scores (mean scores across all three criteria). RQ: review questions: RQ1: Does the knowledge about the biology of B. thuringiensis and its action towards organisms (target and non-target) raise any new questions in relation to the risk assessment of CRY toxins produced by GM plants? RQ2: Does the knowledge about the mode of action of CRY toxins at the molecular level pose any issues for the risk assessment of CRY toxins produced by GMPs? RQ3: How is the phylogenetic relationship, related to mode of action and specificity, between CRY toxins? RQ4: What is the current evidence base to conclude on whether the effects of combined, stacked or pyramided CRY toxins, as produced by stacked GMPs, will be additive, synergistic or antagonistic?

(Reproduced with permission from [19])

No relative weighting of the different criteria was imposed a-priori, so each review team was essentially free to weight the scores for the individual criteria which allowed considering specifics of their thematic. Moreover, on top of stakeholder priorities and comments, the review teams also needed to consider time, resource, and contractual obligations. All review teams were, however, asked to provide short responses outlining their rationale and how they arrived at the final set of review questions.

As a result, 14 of the 69 proposed candidate review questions were selected and decisions pondered whether they would be pursued as systematic reviews or maps. In the process, and based on the additional stakeholder comments, four review questions on health impacts were revised again, and 27 review questions on micro-level socio-economic impacts were merged into one review question for a broad systematic map (final review questions shown in Table 2).
Table 2

Systematic reviews and maps conducted in the context of the GRACE project

GMO impact area

Research question

SR/SM

Ref.a

Health

What are the characteristics of comparative studies of assessment of key chemical crop constituents in GM crops compared to non‐GM crops?

SM

b

What evidence has been collected on the potential toxicity of newly proteins in experimental animals, and what were the characteristics of these studies?

SM

b

What is the evidence for a changed risk of allergic reactions to an allergenic crop after it has been genetically modified?

SM

b

What are the characteristics of repeated‐dose feeding studies with experimental animals receiving whole food or feed products derived from genetically modified (GM) crops and non‐GM counterpart as a control focusing on potential health impacts other than performance?

SM

b

Environmental

Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of non-GM maize?

SR

[25]

What are the effects of the cultivation of GM herbicide tolerant crops on botanical diversity?

SR

[26]

Are population abundances and biomasses of soil invertebrates changed by Bt crops compared with conventional crops?

SR

[27]

Are soil microbial endpoints changed by Bt crops compared with conventional crops?

SR

[28]

How susceptible are different lepidopteran/coleopteran maize pests to Bt-proteins?

SR

[29, 32]

What is the evidence on the inheritance of resistance alleles in populations of lepidopteran/coleopteran maize pest species?

SM

[30]

Socio-economic

What are the socio-economic impacts of genetically modified crops worldwide?

SM

[31]

What is the impact of trade restrictions of GM products in different countries on the competitiveness of different partner countries and corresponding sectors in comparison to a situation where there are no restrictions on GM trade?

SR

c

What is the impact of the introduction of GM crops on the welfare effects in different countries in comparison to a situation where there are restrictions on GM cultivation?

SR

c

What is the impact on GM regulation of different political actors and other drivers in the EU in comparison to the situation in the US?

SM

c

Ref references, SM systematic map, SR systematic review

aProtocols for seven systematic reviews or maps resp. and results of one systematic review were published at the time of writing this article. Preliminary results of the systematic reviews and maps are also available from GRACE Stakeholder Consultation Reports and from the final conference [21, 24]

bPublication of protocols is in preparation

cProtocols will not be published as journal papers as they are considered to be out of scope for the main journals publishing evidence synthesis protocols

The systematic approach for processing stakeholder comments described above was particularly important across these steps. It enabled stakeholders and all interested parties to track how their comments and questions were received and processed by the review teams, if comments suggesting modifications were adopted or not, and if not, the reason for which they were not (fully) adopted. Thereby any interested party could track how stakeholders shaped the choice of topics, scope and wording of the review questions.

Experiences and lessons

The previous sections describe the main characteristics of an evidence synthesis engagement approach developed in response to highly polarized views, lack of trust, and the novelty of evidence synthesis for the respective stakeholder community. This section briefly reflects on some of the experiences and lessons learned (also summarised in Table 3). From the viewpoint of the systematic review community, these considerations are limited due to the lack of completed reviews and maps. At this point, therefore, no conclusions can be derived on how the approach described here ultimately affects the relevance and acceptance of the review findings. From the viewpoint of GMO risk regulation, however, it can already at this stage be concluded that the approach describe here appears to be a very interesting and promising alternative option to render GMO risk research more transparent, inclusive and accountable and, that it is definitely worth to be further explored.
Table 3

Overview of experiences and recommendations for evidence synthesis in controversial contexts

Measures

Experiences—assessment by the authors

Recommendations

Openness and inclusiveness

 Broad stakeholder invitation—any interested stakeholder can participate

The absence of participant selection prevented conflicts about the composition of the stakeholder group and contributed to trust building

This approach needs supporting measures to ensure that interested stakeholder representatives can participate in all subsequent steps (e.g. travel support, smaller stakeholder groups)

 Multiple ways of engagement (workshops, written consultations)

Online written consultations were mainly used to introduce issues not mentioned in or to clarify or emphasize discussion topics of the face-to-face consultations

This set-up is useful for clarifying issues and views and for bringing up issues not addressed during face-to-face meetings, e.g. due to lack of time. It does, however, not help in engaging individuals unable to participate in the face-to-face consultations

 Familiarizing participants with the evidence synthesis process

This measure is very important for a meaningful engagement. This works better if the same representatives of a stakeholder organisation participate in the entire consultation process

The engagement process must be tailored to the level of knowledge of and experience with evidence synthesis

Measures need to be in place to ensure that the same individuals can contribute over the entire engagement process (e.g. providing travel support and opting for smaller stakeholder groups)

Responsiveness

 Procedure for written responses to stakeholder comments

The measure was extensively used by stakeholders

The procedure resulted in a systematic and transparent consideration of stakeholder comments by review teams and allowed for transparency of how stakeholder inputs shaped the process

Time and resource needs are significant and therefore preclude routine application. The procedure appears suitable for sensitive or controversial topics and smaller scale evidence synthesis endeavours

 Stakeholder can influence topic selection and scope

It was challenging to balance stakeholder needs with other project requirements. Many such requirements were contractual (timeline and funding) and therefore given priority

Ultimately decisions about modifications of review questions and process need to stay with and be clearly accountable to the review team(s). In turn the review team(s) should make transparent the reasons for their choices

Transparency

 Providing draft plans and preliminary results for stakeholder review

This measure was appreciated by stakeholders whereas review team members were sometimes reluctant to disclose preliminary plans or results. Participants considered it important for open discussions and trust building

Measures described above are more meaningful and credible if linked to extensive transparency requirements. Ensuring transparency is time consuming so adequate time needs to be planned in advance

 Providing detailed documentation of consultations, stakeholder comments, team responses and how stakeholder comments shaped the process

This measure was appreciated by stakeholders and considered important for open discussions and trust building

Openness and inclusiveness

The evidence synthesis community frequently highlights the importance of stakeholder involvement in the planning stage of systematic reviews and maps. Review questions should be jointly developed with stakeholders [12] but little reflection is available on what approaches have been used and what have been the experiences. For the GMO impact community, stakeholder involvement in the planning stages of research and assessment is a rare endeavour. Not surprisingly, the GRACE opportunity for upstream involvement was positively perceived by participants with the development and prioritization process of the review questions considered as most relevant step.

Broadly circulated invitations at the beginning of the process (step 1) and the absence of participant selection encouraged a total of 95 stakeholder representatives to participate in one or more steps of the process. It also helped to establish and maintain an atmosphere of openness which is helpful for trust building. The only exception applied to journalists who were excluded from workshops to allow participants to speak freely.

Stakeholder balance was well achieved in all steps with the exception of the written consultation on the review protocols: CSO representatives were provided with the protocol but did not comment. Stakeholder balance across the process is shown in Fig. 4.
Figure 4
Fig. 4

Stakeholder balance for each engagement step. N number of stakeholder participants in each step

Openness measures, however, brought in additional challenges: the consecutive consultation steps in the planning process were conceptualized as a step-by-step learning process, hence, the consultation processes in step 2 (prioritizing review questions) and 3 (on draft review protocols) were limited to those participating in step 1. Still, the number of participants was progressively declining and, more importantly, there was considerable fluctuation among individual participations. Only nine of the 41 stakeholder individuals participating in step 1 continued to step 2.

Possible reasons for this participation pattern, as suggested by the authors, are: the high and still increasing number of stakeholder events on GMO issues; the relevance of evidence synthesis for many stakeholders is still not fully clear; the absence of resources to support participation of stakeholders; the lack of familiarity with evidence synthesis (procedure, strengths and weaknesses); the combination of health, environmental and socio-economic topics into the same consultation workshops made it difficult for some people to follow as some topics were outside of their area of expertise (in organisations like competent authorities, industry and research, health, environmental and socio-economic topics are typically covered by different persons); stakeholders felt overloaded with the number of tasks and documents presented to them.

To mitigate such risks, additional measures have to be considered to enable the same individuals to follow the entire process, e.g. smaller groups, more intensive training in evidence synthesis methods and tools, and travel support for stakeholder participants.

Responsiveness

Overall, the GRACE stakeholder consultations led to intense interactions in the course of 2-day workshops and more than 520 written stakeholder comments and team responses. The established procedure to systematically discuss and respond in writing to all stakeholder inputs inspired internal discussions of the review teams. As described in previous sections, stakeholder comments effectively shaped the prioritization, scope and phrasing of the review questions and thereby improved the quality and relevance of the review plans.

On the other hand, stakeholder suggestions sometimes conflicted with resource, time and other requirements of the GRACE project. Occasionally, stakeholder suggestions were contradicting each other, thus it was very important that ultimate decisions remained with the review team.

Review teams, however, needed to be transparent with which inputs they have considered or not, and why. Suggestions not taken sometimes led to disappointment or frustration from stakeholder participants, this being indicative of the need to carefully balance stakeholder expectations and the flexibility to accommodate inputs.

Transparency

Transparency measures were generally appreciated by stakeholders but sometimes resulted in a reluctance of scientists to share preliminary and draft plans with a broader stakeholder group. The reason was the hesitation of some scientists to expose their output to a thorough scrutiny by a polarized stakeholder community at a point in time when such results were considered preliminary or even estimative. Moreover, some scientists pursuing a typical academic career had difficulties to see what they could gain from an ‘extended peer review’ at this stage.

The procedure for tracing and making transparent how stakeholder inputs were perceived and processed by review teams was—to the best knowledge of the authors—used for the first time in evidence synthesis and was positively received by stakeholder participants and observers [36]. The detailed documentation of all engagement steps also provides a unique evidence base for any ex-post analysis of stakeholder views and their impacts on the project.

The extensive transparency requirements allowed better identification and understanding of divergent views and assessments and were important to establish and maintain the atmosphere of openness despite long standing lack of trust among some stakeholders.

Setting an example for other controversial regulatory science topics?

The consultative approach described here offers interesting advantages if operating in the context of long-standing controversies which often goes together with polarised views and lack of trust. In such contexts the substantial time and resource requirements from stakeholder participants and review teams would also be well justified.

The approach allows to systematically capture and make transparent stakeholder and review team views and priorities. It also provides elements for openly and transparently engaging with stakeholders in planning stages of the review without the ultimate obligation to arrive at a consensus—something which is very difficult to achieve in case of long standing controversies. Still, as shown in this paper, the procedure can help to improve the quality and relevance of review topics and plans.

Another challenge highlighted in this paper is how to operate in contexts with little or no prior experience with evidence synthesis. Although introductions and a training into evidence synthesis were provided this was by no way sufficient. Many stakeholder representatives participating in consultation workshops did not participate in the training and had superficial understandings of the methodology. Moreover, the measures in place to provide openness resulted in participation patterns making step-by-step knowledge generation difficult.

Drawing on this experience the approach could work better in the course of smaller scale endeavours, focussing on a smaller number of review questions in a more homogenous field of research, providing more intense training in systematic review methodology as well as providing resources to stakeholders to encourage continuing participation and tackle attrition.

Abbreviations

CADIMA: 

Central Access Database for Impact Assessment of Crop Genetic Improvement Technologies

CEE: 

Collaboration for Environmental Evidence

Bt-crops: 

genetically modified crops producing an insecticide following genetic transfer of a gene from a bacterial species Bacillus thuringiensis

GM: 

genetic modification

GMO: 

genetically modified organisms

GRACE: 

GMO risk assessment and communication of evidence

RQ: 

review question

SM: 

systematic maps

SR: 

systematic reviews

Declarations

Authors’ contributions

AS let the work package on stakeholder engagement and was the principle researcher for the stakeholder engagement concept. He also drafted this manuscript with the help of MR. All authors contributed to the manuscript. AS, GA, GS, SK, MR and PR jointly planned, organised and documented the activities described in the paper. Online prioritization was conducted by GA and AS. All authors read and approved the final manuscript.

Acknowledgements

The authors are acknowledging in particular the 95 stakeholders from 17 EEA Member States, the EU-level, the USA and international organisations that participated in workshops, online questionnaires, and written consultations in the course of the GRACE project. All organisations that participated and contributed at each step are explicitly acknowledged in the stakeholder consultation reports available at http://www.grace-fp7.eu/.

Furthermore, the authors are grateful to all other members of the GRACE team and Advisory Board, in particular to Wendy Craig, Jaqueline Garcia-Yi, Maria Garrone, Karen Hofreiter, Gijs Kleter, Klaus Minol, Jeremy Sweet, and Geoff Frampton, for their support in implementing and maintaining the project goals in terms of inclusiveness, transparency, and responsiveness over the entire project period.

Competing interests

The authors declare that they have no competing interests.

Availability of data and materials

The datasets of the systematic reviews and systematic maps referred to in the article are available or will be made available at https://www.cadima.info/. Detailed documentations of the stakeholder engagement steps including the development and selection of review questions are provided at http://www.grace-fp7.eu/.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Funding

The research described in this article was funded by the European Commission 7th Framework Programme in the context of the large collaborative project GRACE (Grant Agreement No 311957; http://www.grace-fp7.eu).

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Graz University of Technology and Alpen-Adria Universität Klagenfurt, Wien-Graz, Graz, Austria
(2)
Alpen-Adria Universität Klagenfurt, Wien-Graz, Graz, Austria
(3)
Julius Kühn-Institut, Quedlinburg, Germany
(4)
Perseus BVBA, Sint-Martens-Latem, Belgium

References

  1. Devos Y, Sanvido O, Tait J, Raybould A. Towards a more open debate about values in decision-making on agricultural biotechnology. Transgenic Res. 2014;23:933–43.View ArticleGoogle Scholar
  2. Séralini GE, Clair E, Mesnage R, Gress S, Defarge N, Malatesta M, Hennequin D, de Vendômois JS. Republished study: long-term toxicity of a Roundup herbicide and a roundup-tolerant genetically modified maize. Environ Sci Eur. 2014;26:14.View ArticleGoogle Scholar
  3. EFSA. Review of the Séralini et al. (2012) publication on a 2-year rodent feeding study with glyphosate formulations and GM maize NK603 as published online on 19 September 2012 in Food and Chemical Toxicology. EFSA J. 2012;10(10):2910.View ArticleGoogle Scholar
  4. Schmidt JEU, Braun CU, Whitehouse LP, Hilbeck A. Effects of activated Bt transgene products (Cry1Ab, Cry3Bb) on immature stages of the ladybird Adalia bipunctata in laboratory ecotoxicity testing. Arch Environ Contam Toxicol. 2009;56:221–8.View ArticleGoogle Scholar
  5. Alvarez-Alfageme F, Bigler F, Romeis J. Laboratory toxicity studies demonstrating no adverse effects of Cry1Ab and Cry3Bb1 to larvae of Adalia bipunctata (Coleoptera: coccinellidae): the importance of study design. Transgenic Res. 2010;20:467–79.View ArticleGoogle Scholar
  6. Rauschen S. A case of ‘pseudo science’? A study claiming effects of the Cry1Ab protein on larvae of the two-spotted ladybird is reminiscent of the case of the green lacewing. Transgenic Res. 2010;19:13–6.View ArticleGoogle Scholar
  7. Ricroch A, Bergé JB, Kuntz M. Is the German suspension of Mon810 maize cultivation scientifically justified? Transgenic Res. 2010;19:1–12.View ArticleGoogle Scholar
  8. Séralini GE, Mesnage R, Defarge N, de Vendômois JS. Conflicts of interests, confidentiality and censorship in health risk assessment: the example of a herbicide and a GMO. Environ Sci Eur. 2014;26:13.View ArticleGoogle Scholar
  9. GRACE Project Website http://www.grace-fp7.eu/. 2016. Accessed 20 Dec 2016.
  10. http://www.cochrane.org/, http://www.campbellcollaboration.org/, http://www.environmentalevidence.org/. Accessed 5 Jan 2018.
  11. EFSA. European food safety authority; application of systematic review methodology to food and feed safety assessments to support decision making. EFSA J. 2010;8:1637.View ArticleGoogle Scholar
  12. CEE (Collaboration for Environmental Evidence). Centre for Evidence-Based Conservation. Guidelines for Systematic Review in Environmental Management. Version 4.2. Environmental Evidence. 2013. www.environmentalevidence.org/Documents/Guidelines/Guidelines4.2.pdf. Accessed 20 Dec 2016.
  13. Kohl C, Frampton G, Damgaard CF, Wilhelm R, Schiemann J. Draft Guidelines for Good Review Practice in GMO Impact Assessment. GRACE Project Report. 2013. http://www.grace-fp7.eu/sites/default/files/DRAFT%20Info%20stakeholder%20SR_WP8_GA-final.pdf. Accessed 5 Jan 2018.
  14. Von Schomberg R. Prospects for technology assessment in a framework of responsible research and innovation’ in: Technikfolgen abschätzen lehren: Bildungspotenziale transdisziplinärer Methode, Wiesbaden: Springer; 2011. p 39.Google Scholar
  15. Owen R, Macnaghten P, Stilgoe J. Responsible research and innovation: from science in society to science for society, with society. Sci Public Policy. 2012;39(6):751.View ArticleGoogle Scholar
  16. Owen R, Stilgoe J, Macnaghten P, Gorman M, Fisher E, Guston D. A framework for responsible innovation. In: Owen R, Bessant J, Heintz M, editors. Responsible innovation: managing the responsible emergence of science and innovation in society. John Wiley & Sons Ltd, Chichester, UK; 2013, p 27–50. https://doi.org/10.1002/9781118551424.ch2.
  17. Smets G, Spök A et al. Assessing the Evidence of Health, Environmental and Socio-Economic Impacts of GMO—GRACE Stakeholder Consultation on Good Review Practice in GMO Impact Assessment: Part 1: Overall Process and Review Questions1, GRACE Report. 2013. http://www.grace-fp7.eu/en/content/reports-study-plans-consultation-documents. Accessed 20 Dec 2016.
  18. Adduci G, Spök A. (eds). GRACE Stakeholder Consultation on Good Review Practice in GMO Impact Assessment: Part 2: Overall Process and Review Questions—Responses to Stakeholder Comments, GRACE Report. 2014. http://www.grace-fp7.eu/en/content/reports-study-plans-consultation-documents. Accessed 20 Dec 2016.
  19. Adduci G, Spök A. GRACE Stakeholder Consultation on Good Review Practice in GMO Impact Assessment: Part 3: Stakeholder Priorities for Review Questions, GRACE Report. 2014. http://www.grace-fp7.eu/en/content/reports-study-plans-consultation-documents. Accessed 20 Dec 2016.
  20. Adduci G, Spök A. GRACE Stakeholder Consultation on Good Review Practice in GMO Impact Assessment: Part 4: Draft Protocols for Systematic Reviews and Evidence Maps, GRACE Report. 2014. http://www.grace-fp7.eu/en/content/reports-study-plans-consultation-documents. Accessed 20 Dec 2016.
  21. Racovita M, Adduci G, Karner S, Spök A. Environmental and Socio-Economic Impacts of GMOs. Stakeholder Consultation on the Results of the GRACE Systematic Reviews and Evidence Maps and on Their Role in Future GMO Impact Research and Assessment, GRACE Report. 2015. http://www.grace-fp7.eu/en/content/reports-study-plans-consultation-documents. Accessed 20 Dec 2016.
  22. Kohl C, Frampton G, Sweet J, Spök A, Haddaway NR, Wilhelm R, Unger S, Schiemann J. Can systematic reviews inform GMO risk assessment and risk management? Front Bioeng Biotechnol. 2015. https://doi.org/10.3389/fbioe.2015.00113.Google Scholar
  23. GRACE: Conclusions and recommendations on animal feeding trials and alternative approaches and on the use of systematic reviews and evidence maps for GMO impact assessment. 2015 http://www.grace-fp7.eu/. Accessed 20 Dec 2016.
  24. GRACE. Presentations. GRACE Final Conference. 2015. http://www.grace-fp7.eu/sites/default/files/Presentations-GFC-Nov2015-final.pdf. Accessed 20 Dec 2016.
  25. Meissle M, Naranjo SE, Kohl C, Riedel J, Romeis J. Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of non-GM maize? A systematic review protocol. Environ Evid. 2014;3:7.View ArticleGoogle Scholar
  26. Sweet J, Kostov K. What are the effects of the cultivation of GM herbicide tolerant crops on botanical diversity? A systematic review protocol. Environ Evid. 2014;3:8.View ArticleGoogle Scholar
  27. Kostov K, Damgaard CF, Hendriksen NB, Sweet JB, Krogh PH. Are population abundances and biomasses of soil invertebrates changed by Bt crops compared with conventional crops? Environ Evid. 2014;3:10.View ArticleGoogle Scholar
  28. Kostov K, Krogh PH, Damgaard CF, Sweet JB, Hendriksen NB. Are soil microbial endpoints changed by Bt crops compared with conventional crops? Environ Evid. 2014;3:11.View ArticleGoogle Scholar
  29. Gathmann A, Priesnitz KU. How susceptible are different lepidopteran/coleopteran maize pests to Bt-proteins: a systematic review protocol. Environ Evid. 2014;3:12.View ArticleGoogle Scholar
  30. Gathmann A, Priesnitz KU. What is the evidence on the inheritance of resistance alleles in populations of lepidopteran/coleopteran maize pest species: a systematic map protocol. Environ Evid. 2014;3:13.View ArticleGoogle Scholar
  31. Garcia-Yi J, Lapikanonth T, Vionita H, Vu H, Yang S, Zhong Y, Li Y, Nagelschneider V, Schlindwein B, Wesseler J. What are the socio-economic impacts of genetically modified crops worldwide? A systematic map protocol. Environ Evid. 2014;3:24.View ArticleGoogle Scholar
  32. Gathmann A, Vaasen A, Priesnitz KU. Baseline susceptibility of different European lepidopteran and coleopteran pests to Bt proteins expressed in Bt maize: a systematic review. Environ Evid. 2016;5:27.View ArticleGoogle Scholar
  33. CADIMA database. https://www.cadima.info/. 2016. Accessed 20 Dec 2016.
  34. O’Connor AM, Lovei GL, Eales J, Frampton GK, Glanville J, Pullin AS, Sargeant J. Implementation of SRs in EFSA scientific outputs workflow. EFSA Supporting Publications 2012. EN-367. http://www.efsa.europa.eu/de/supporting/doc/367e.pdf. Accessed 20 Dec 2016.
  35. Clavisi O, Bragge P, Tavender E, Turner T, Gruen RL. Effective stakeholder participation in setting research priorities using a Global Evidence Mapping approach. J Clin Epidemiol. 2013;66(5):496–502.View ArticleGoogle Scholar
  36. COGEM. Aandachtspunten en valkuilen stakeholdersbetrokkenheid bij onderzoek naar controversiële onderwerpen (CGM/170111-03) 2017. http://www.cogem.net/index.cfm/nl/publicaties/publicatie/aandachtspunten-en-valkuilen-stakeholdersbetrokkenheid-bij-onderzoek-naar-controversiele-onderwerpen; inofficial translation: COGEM Points of attention and pitfalls related to stakeholder involvement in research on controversial topics (CGM/170111-03) 2017. https://www.g-twyst.eu/files/Thirdparties/Translation170111-03signalerendebriefstakeholderparticipatieonderzoekcontroversieleonderwerpen170117.pdf. Accessed 18 Feb 2017.

Copyright

© The Author(s) 2018

Advertisement