How to best assess potential health, environmental and other impacts of genetically modified organisms (GMOs) and how to interpret the resulting evidence base have been long-standing controversial issues in the EU. As a response, transparency and inclusiveness became a major focus of regulatory science activities in the GMO impact area. Nevertheless, nearly three decades of controversies resulted in a heavily polarized policy environment, calling for further efforts. Against this backdrop the EU funded project GRACE explored the value of evidence synthesis approaches for GMO impact assessment and developed an evidence synthesis framework with a strong emphasis on openness, stakeholder engagement, transparency, and responsiveness to tackle regulatory science challenges. This framework was tested and implemented in the course of 14 systematic reviews or maps conducted on selected review questions spanning potential health, environmental, and socioeconomic impacts of GMOs. An inclusive development and prioritisation of review questions is of key importance in evidence synthesis as it helps to provide a better link between stakeholder demands and concerns and policy relevant outcomes. This paper, therefore, places a particular focus on the stakeholder involvement strategy developed and experiences gathered during this particular step in the course of the GRACE project. Based on this experience, possible lessons for future engagement exercises in highly controversial fields of regulatory science are discussed.
Evaluating potential health, environmental and socioeconomic impacts of genetically modified (GM) crops has been a dynamic field of research since the early 1990s. Interpreting the resulting primary data has frequently been a subject of controversial debates augmented by the complexity and the diversity of test designs and the multitude of endpoints under investigation  as well as by contradicting results. Reviewing secondary data by academia and regulatory committees has frequently led to discussions about, among other, studies not (appropriately) considered, relative weight attached to data generated by different methods, bias, lack of transparency and to divergent conclusions [2,3,4,5,6,7,8].
Against this backdrop the EU funded research project GRACE (GMO Risk Assessment and Communication of Evidence) set out to explore the use of evidence synthesis approaches to assess and synthesize exiting evidence on potential health, environmental and socio-economic impacts of GM crops . At the time of the planning and start of this project (2011–2012) systematic reviews and maps had sparingly been applied to GMO impact research.
GRACE had a twofold aim: (i) to explore the suitability of systematic reviews and maps in GMO impact research and assessment, and (ii) to assess the available evidence on selected research questions. The results were expected to support evidence-based policy making in GMO impact assessment.
By adapting existing guidance documents [10,11,12] GRACE developed a general framework appropriate for the synthesis of GM crop impact data . Based on this general framework 14 systematic reviews and maps were planned and started in parallel.
A particular emphasis was placed on an active stakeholder involvement going beyond what is generally done in the course of systematic reviews and maps. A multi-step stakeholder engagement approach was developed inspired by two main considerations: (i) the long-standing polarization of the GMO impact debate which is accompanied by a lack of trust among stakeholders and (ii) the concept of Responsible Research and Innovation (RRI) underlying the recent research programmes funded by the European Commission and many national agencies [14,15,16]. The resulting engagement approach, aiming to strengthen the openness, inclusiveness, responsiveness, and transparency of systematic reviews, was implemented for all GRACE reviews and maps in a harmonised way by a team specifically set for the purpose, involving researchers with expertise in natural and social sciences.
According to participant surveys and authors’ experiences, stakeholder participation was perceived to be most productive and important in the development and selection of the review questions. As shown in this paper, stakeholders had a considerable impact on the definition and selection of research questions and, thereby, on the scope of synthesis results. These steps are, therefore, the focus of this paper.
Characteristics of the GRACE stakeholder engagement approach for planning the reviews and maps
An overview of the stakeholder engagement approach is provided in Fig. 1.
Participation in stakeholder workshops was open for all interested stakeholder representatives following a broad invitation circulated to some 500 stakeholder contacts encompassing a broad range of, among others, competent authorities, industry, civil society organisations, professional organisations and researchers. Subsequent steps mainly involved workshop participants.
Despite limited resources, considerable attempts were made to have the key stakeholder groups represented across all steps.
As a first step, the overall framework and preliminary review questions were discussed in a 2-day workshop. Stakeholders could provide written comments and questions on the workshop discussions but also on issues not discussed there. Written comments were open to a wide audience including, but not limited to, workshop participants.
The resulting lists of the candidate review questions were prioritized in a second step by stakeholders using an online questionnaire, helping thus the project team identify a final set of review questions.
In the third step review protocols were developed and subjected, as drafts, to stakeholder comments in the course of a written consultation. Following a revision of the protocols prompted by stakeholder comments, they were published or prepared for publication as peer-reviewed journal papers.
In the fourth step preliminary results of the individual reviews as well as draft general conclusions and recommendations were presented and discussed in the course of another 2 day stakeholder workshop and again subjected to written comments.
Stakeholders participating in the workshops as well as those provided with consultation materials could provide comments to the review team, which were then collected, categorised and subjected to the individual teams overview through the project’ s internal clearing house mechanism. Review teams discussed all comments, revised the review questions and developed written point-by-point responses to all comments and questions.
Workshop participants were provided with draft documents and presentations before the workshop meetings. Each step was documented in a detailed stakeholder consultation report which covered workshop inputs and discussions, survey results, written stakeholder questions and comments as well as review team responses. Workshop reports documenting discussions were circulated as drafts to all participants in order to ensure that discussions and viewpoints were accurately depicted. The workshops, the written stakeholder comments, and the review teams’ responses are documented in detail in a series of consultations reports available [17,18,19,20,21] on the project website .
To better understand how the GRACE approach was perceived by participants, their views were gathered via phone interviews as well as via online and paper questionnaires.
Status of GRACE systematic reviews and maps
The characteristics described above also accompanied the stakeholder consultations on the preliminary results of the individual reviews and maps (not shown as they are beyond the scope of this paper). Furthermore, GRACE developed general conclusions and recommendations on the use and value of evidence synthesis approaches in GMO impact assessment [22,23,24] which were also included in the scope of the consultations. By the time of the submission of this paper, 7 of the 14 review/map protocols [25,26,27,28,29,30,31] and the results of one systematic review  have been published as peer reviewed open-access papers and 10 review/map protocols and the published systematic review are available on the Open Access database CADIMA .
Development and prioritization of review questions
The main steps in the development and prioritization of review questions are depicted in Fig. 1—indicating also the methodology used. The manner in which this process influenced the evolution of review questions is indicated in Table 1 and illustrated in Fig. 2.
Proposing and discussing review questions
In the first step a total of 68 preliminary review questions covering the three fields of GMO impact research were subjected to a consultation with 41 stakeholders. During this step the focus was on understanding and discussing the general characteristics of systematic reviews and maps, the potential relevance of this method for GMO impact research and assessment, the conceptual frameworks for the review questions, and the preliminary review questions.
Based on the workshop discussions and more than 170 written stakeholder comments received after the workshop, the project team modified the review questions considerably in terms of subject, scope, wording and numbers (summarised in Table 1). Of a total of 68 review questions, 23 were revised, 11 dropped and 13 new questions added. For the health and environmental impacts work streams this resulted in a reduction of the number of review questions. For the socio-economic impacts work stream it led to a broadening of the scope and to adding more review questions.
The resulting 69 candidate review questions were then prioritized by stakeholders in the next step.
Developing criteria and a process for prioritizing review questions
The method and criteria used were tailored to accommodate (i) the specific challenges of dealing with a large number of review questions in three very different scientific and policy contexts in parallel, (ii) the fact that this prioritization was done by stakeholders in a very polarized policy environment, (iii) the constraints of a relatively inflexible research project with tight timelines and resource limitations, and (iv) the exploratory nature of the project in applying evidence synthesis to GMO impact research.
The criteria used were (i) importance for impact assessment, (ii) the existence of expert controversy, and (iii) degree of public awareness (inspired by O’Connner et al.  and Clavesi et al. ). These three criteria refer to three important dimensions, which also act as three different contexts in the GMO debate: scientific, regulatory, and public dimension. A scoring system (1–5) was used with the option for adding comments allowing for a more nuanced feed-back.
Prioritization of the candidate review questions
The stakeholders participating in the previous consultation step plus those who expressed a particular interest (n = 55) were asked to score each of the 69 candidate review questions in each of the three criteria by using an electronic questionnaire (LimeSurvey).
The respondent rate of the questionnaire was 34.5% (health and environmental impacts) and 29.1% (socio-economic impacts) respectively. Aside from scores, stakeholders provided 74 additional comments. The resulting data set was analysed using SPSS and Excel. Mean scores for each criterion as well as aggregated scores (sum of mean scores of each criterion—assuming an equal relative weight of each criterion) were calculate for each candidate review question and where discussed and considered along with additional stakeholder comments by the three project teams (health, environmental and socioeconomic). An example of how stakeholder scores were reported to the review teams is provided in Fig. 3.
No relative weighting of the different criteria was imposed a-priori, so each review team was essentially free to weight the scores for the individual criteria which allowed considering specifics of their thematic. Moreover, on top of stakeholder priorities and comments, the review teams also needed to consider time, resource, and contractual obligations. All review teams were, however, asked to provide short responses outlining their rationale and how they arrived at the final set of review questions.
As a result, 14 of the 69 proposed candidate review questions were selected and decisions pondered whether they would be pursued as systematic reviews or maps. In the process, and based on the additional stakeholder comments, four review questions on health impacts were revised again, and 27 review questions on micro-level socio-economic impacts were merged into one review question for a broad systematic map (final review questions shown in Table 2).
The systematic approach for processing stakeholder comments described above was particularly important across these steps. It enabled stakeholders and all interested parties to track how their comments and questions were received and processed by the review teams, if comments suggesting modifications were adopted or not, and if not, the reason for which they were not (fully) adopted. Thereby any interested party could track how stakeholders shaped the choice of topics, scope and wording of the review questions.
Experiences and lessons
The previous sections describe the main characteristics of an evidence synthesis engagement approach developed in response to highly polarized views, lack of trust, and the novelty of evidence synthesis for the respective stakeholder community. This section briefly reflects on some of the experiences and lessons learned (also summarised in Table 3). From the viewpoint of the systematic review community, these considerations are limited due to the lack of completed reviews and maps. At this point, therefore, no conclusions can be derived on how the approach described here ultimately affects the relevance and acceptance of the review findings. From the viewpoint of GMO risk regulation, however, it can already at this stage be concluded that the approach describe here appears to be a very interesting and promising alternative option to render GMO risk research more transparent, inclusive and accountable and, that it is definitely worth to be further explored.
Openness and inclusiveness
The evidence synthesis community frequently highlights the importance of stakeholder involvement in the planning stage of systematic reviews and maps. Review questions should be jointly developed with stakeholders  but little reflection is available on what approaches have been used and what have been the experiences. For the GMO impact community, stakeholder involvement in the planning stages of research and assessment is a rare endeavour. Not surprisingly, the GRACE opportunity for upstream involvement was positively perceived by participants with the development and prioritization process of the review questions considered as most relevant step.
Broadly circulated invitations at the beginning of the process (step 1) and the absence of participant selection encouraged a total of 95 stakeholder representatives to participate in one or more steps of the process. It also helped to establish and maintain an atmosphere of openness which is helpful for trust building. The only exception applied to journalists who were excluded from workshops to allow participants to speak freely.
Stakeholder balance was well achieved in all steps with the exception of the written consultation on the review protocols: CSO representatives were provided with the protocol but did not comment. Stakeholder balance across the process is shown in Fig. 4.
Openness measures, however, brought in additional challenges: the consecutive consultation steps in the planning process were conceptualized as a step-by-step learning process, hence, the consultation processes in step 2 (prioritizing review questions) and 3 (on draft review protocols) were limited to those participating in step 1. Still, the number of participants was progressively declining and, more importantly, there was considerable fluctuation among individual participations. Only nine of the 41 stakeholder individuals participating in step 1 continued to step 2.
Possible reasons for this participation pattern, as suggested by the authors, are: the high and still increasing number of stakeholder events on GMO issues; the relevance of evidence synthesis for many stakeholders is still not fully clear; the absence of resources to support participation of stakeholders; the lack of familiarity with evidence synthesis (procedure, strengths and weaknesses); the combination of health, environmental and socio-economic topics into the same consultation workshops made it difficult for some people to follow as some topics were outside of their area of expertise (in organisations like competent authorities, industry and research, health, environmental and socio-economic topics are typically covered by different persons); stakeholders felt overloaded with the number of tasks and documents presented to them.
To mitigate such risks, additional measures have to be considered to enable the same individuals to follow the entire process, e.g. smaller groups, more intensive training in evidence synthesis methods and tools, and travel support for stakeholder participants.
Overall, the GRACE stakeholder consultations led to intense interactions in the course of 2-day workshops and more than 520 written stakeholder comments and team responses. The established procedure to systematically discuss and respond in writing to all stakeholder inputs inspired internal discussions of the review teams. As described in previous sections, stakeholder comments effectively shaped the prioritization, scope and phrasing of the review questions and thereby improved the quality and relevance of the review plans.
On the other hand, stakeholder suggestions sometimes conflicted with resource, time and other requirements of the GRACE project. Occasionally, stakeholder suggestions were contradicting each other, thus it was very important that ultimate decisions remained with the review team.
Review teams, however, needed to be transparent with which inputs they have considered or not, and why. Suggestions not taken sometimes led to disappointment or frustration from stakeholder participants, this being indicative of the need to carefully balance stakeholder expectations and the flexibility to accommodate inputs.
Transparency measures were generally appreciated by stakeholders but sometimes resulted in a reluctance of scientists to share preliminary and draft plans with a broader stakeholder group. The reason was the hesitation of some scientists to expose their output to a thorough scrutiny by a polarized stakeholder community at a point in time when such results were considered preliminary or even estimative. Moreover, some scientists pursuing a typical academic career had difficulties to see what they could gain from an ‘extended peer review’ at this stage.
The procedure for tracing and making transparent how stakeholder inputs were perceived and processed by review teams was—to the best knowledge of the authors—used for the first time in evidence synthesis and was positively received by stakeholder participants and observers . The detailed documentation of all engagement steps also provides a unique evidence base for any ex-post analysis of stakeholder views and their impacts on the project.
The extensive transparency requirements allowed better identification and understanding of divergent views and assessments and were important to establish and maintain the atmosphere of openness despite long standing lack of trust among some stakeholders.
Setting an example for other controversial regulatory science topics?
The consultative approach described here offers interesting advantages if operating in the context of long-standing controversies which often goes together with polarised views and lack of trust. In such contexts the substantial time and resource requirements from stakeholder participants and review teams would also be well justified.
The approach allows to systematically capture and make transparent stakeholder and review team views and priorities. It also provides elements for openly and transparently engaging with stakeholders in planning stages of the review without the ultimate obligation to arrive at a consensus—something which is very difficult to achieve in case of long standing controversies. Still, as shown in this paper, the procedure can help to improve the quality and relevance of review topics and plans.
Another challenge highlighted in this paper is how to operate in contexts with little or no prior experience with evidence synthesis. Although introductions and a training into evidence synthesis were provided this was by no way sufficient. Many stakeholder representatives participating in consultation workshops did not participate in the training and had superficial understandings of the methodology. Moreover, the measures in place to provide openness resulted in participation patterns making step-by-step knowledge generation difficult.
Drawing on this experience the approach could work better in the course of smaller scale endeavours, focussing on a smaller number of review questions in a more homogenous field of research, providing more intense training in systematic review methodology as well as providing resources to stakeholders to encourage continuing participation and tackle attrition.
Central Access Database for Impact Assessment of Crop Genetic Improvement Technologies
Collaboration for Environmental Evidence
genetically modified crops producing an insecticide following genetic transfer of a gene from a bacterial species Bacillus thuringiensis
genetically modified organisms
GMO risk assessment and communication of evidence
Devos Y, Sanvido O, Tait J, Raybould A. Towards a more open debate about values in decision-making on agricultural biotechnology. Transgenic Res. 2014;23:933–43.
Séralini GE, Clair E, Mesnage R, Gress S, Defarge N, Malatesta M, Hennequin D, de Vendômois JS. Republished study: long-term toxicity of a Roundup herbicide and a roundup-tolerant genetically modified maize. Environ Sci Eur. 2014;26:14.
EFSA. Review of the Séralini et al. (2012) publication on a 2-year rodent feeding study with glyphosate formulations and GM maize NK603 as published online on 19 September 2012 in Food and Chemical Toxicology. EFSA J. 2012;10(10):2910.
Schmidt JEU, Braun CU, Whitehouse LP, Hilbeck A. Effects of activated Bt transgene products (Cry1Ab, Cry3Bb) on immature stages of the ladybird Adalia bipunctata in laboratory ecotoxicity testing. Arch Environ Contam Toxicol. 2009;56:221–8.
Alvarez-Alfageme F, Bigler F, Romeis J. Laboratory toxicity studies demonstrating no adverse effects of Cry1Ab and Cry3Bb1 to larvae of Adalia bipunctata (Coleoptera: coccinellidae): the importance of study design. Transgenic Res. 2010;20:467–79.
Rauschen S. A case of ‘pseudo science’? A study claiming effects of the Cry1Ab protein on larvae of the two-spotted ladybird is reminiscent of the case of the green lacewing. Transgenic Res. 2010;19:13–6.
Von Schomberg R. Prospects for technology assessment in a framework of responsible research and innovation’ in: Technikfolgen abschätzen lehren: Bildungspotenziale transdisziplinärer Methode, Wiesbaden: Springer; 2011. p 39.
Owen R, Macnaghten P, Stilgoe J. Responsible research and innovation: from science in society to science for society, with society. Sci Public Policy. 2012;39(6):751.
Owen R, Stilgoe J, Macnaghten P, Gorman M, Fisher E, Guston D. A framework for responsible innovation. In: Owen R, Bessant J, Heintz M, editors. Responsible innovation: managing the responsible emergence of science and innovation in society. John Wiley & Sons Ltd, Chichester, UK; 2013, p 27–50. https://doi.org/10.1002/9781118551424.ch2.
GRACE: Conclusions and recommendations on animal feeding trials and alternative approaches and on the use of systematic reviews and evidence maps for GMO impact assessment. 2015 http://www.grace-fp7.eu/. Accessed 20 Dec 2016.
Meissle M, Naranjo SE, Kohl C, Riedel J, Romeis J. Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of non-GM maize? A systematic review protocol. Environ Evid. 2014;3:7.
Gathmann A, Priesnitz KU. What is the evidence on the inheritance of resistance alleles in populations of lepidopteran/coleopteran maize pest species: a systematic map protocol. Environ Evid. 2014;3:13.
Garcia-Yi J, Lapikanonth T, Vionita H, Vu H, Yang S, Zhong Y, Li Y, Nagelschneider V, Schlindwein B, Wesseler J. What are the socio-economic impacts of genetically modified crops worldwide? A systematic map protocol. Environ Evid. 2014;3:24.
AS let the work package on stakeholder engagement and was the principle researcher for the stakeholder engagement concept. He also drafted this manuscript with the help of MR. All authors contributed to the manuscript. AS, GA, GS, SK, MR and PR jointly planned, organised and documented the activities described in the paper. Online prioritization was conducted by GA and AS. All authors read and approved the final manuscript.
The authors are acknowledging in particular the 95 stakeholders from 17 EEA Member States, the EU-level, the USA and international organisations that participated in workshops, online questionnaires, and written consultations in the course of the GRACE project. All organisations that participated and contributed at each step are explicitly acknowledged in the stakeholder consultation reports available at http://www.grace-fp7.eu/.
Furthermore, the authors are grateful to all other members of the GRACE team and Advisory Board, in particular to Wendy Craig, Jaqueline Garcia-Yi, Maria Garrone, Karen Hofreiter, Gijs Kleter, Klaus Minol, Jeremy Sweet, and Geoff Frampton, for their support in implementing and maintaining the project goals in terms of inclusiveness, transparency, and responsiveness over the entire project period.
The authors declare that they have no competing interests.
Availability of data and materials
The datasets of the systematic reviews and systematic maps referred to in the article are available or will be made available at https://www.cadima.info/. Detailed documentations of the stakeholder engagement steps including the development and selection of review questions are provided at http://www.grace-fp7.eu/.
Consent for publication
Ethics approval and consent to participate
The research described in this article was funded by the European Commission 7th Framework Programme in the context of the large collaborative project GRACE (Grant Agreement No 311957; http://www.grace-fp7.eu).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Authors and Affiliations
Graz University of Technology and Alpen-Adria Universität Klagenfurt, Wien-Graz, Graz, Austria
Alpen-Adria Universität Klagenfurt, Wien-Graz, Graz, Austria
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Spök, A., Racovita, M., Karner, S. et al. Inclusive development and prioritization of review questions in a highly controversial field of regulatory science.
Environ Evid7, 1 (2018). https://doi.org/10.1186/s13750-017-0113-z