Inclusive development and prioritization of review questions in a highly controversial field of regulatory science

How to best assess potential health, environmental and other impacts of genetically modified organisms (GMOs) and how to interpret the resulting evidence base have been long-standing controversial issues in the EU. As a response, transparency and inclusiveness became a major focus of regulatory science activities in the GMO impact area. Nevertheless, nearly three decades of controversies resulted in a heavily polarized policy environment, calling for further efforts. Against this backdrop the EU funded project GRACE explored the value of evidence synthesis approaches for GMO impact assessment and developed an evidence synthesis framework with a strong emphasis on openness, stakeholder engagement, transparency, and responsiveness to tackle regulatory science challenges. This framework was tested and implemented in the course of 14 systematic reviews or maps conducted on selected review questions spanning potential health, environmental, and socioeconomic impacts of GMOs. An inclusive development and prioritisation of review questions is of key importance in evidence synthesis as it helps to provide a better link between stakeholder demands and concerns and policy relevant outcomes. This paper, therefore, places a particular focus on the stakeholder involvement strategy developed and experiences gathered during this particular step in the course of the GRACE project. Based on this experience, possible lessons for future engagement exercises in highly controversial fields of regulatory science are discussed.


Background
Evaluating potential health, environmental and socioeconomic impacts of genetically modified (GM) crops has been a dynamic field of research since the early 1990s. Interpreting the resulting primary data has frequently been a subject of controversial debates augmented by the complexity and the diversity of test designs and the multitude of endpoints under investigation [1] as well as by contradicting results. Reviewing secondary data by academia and regulatory committees has frequently led to discussions about, among other, studies not (appropriately) considered, relative weight attached to data generated by different methods, bias, lack of transparency and to divergent conclusions [2][3][4][5][6][7][8].
Against this backdrop the EU funded research project GRACE (GMO Risk Assessment and Communication of Evidence) set out to explore the use of evidence synthesis approaches to assess and synthesize exiting evidence on potential health, environmental and socio-economic impacts of GM crops [9]. At the time of the planning and start of this project (2011-2012) systematic reviews and maps had sparingly been applied to GMO impact research.
GRACE had a twofold aim: (i) to explore the suitability of systematic reviews and maps in GMO impact research and assessment, and (ii) to assess the available evidence on selected research questions. The results were expected

Open Access
Environmental Evidence *Correspondence: armin.spoek@tugraz.at 1 Graz University of Technology and Alpen-Adria Universität Klagenfurt, Wien-Graz, Graz, Austria Full list of author information is available at the end of the article to support evidence-based policy making in GMO impact assessment.
By adapting existing guidance documents [10-12] GRACE developed a general framework appropriate for the synthesis of GM crop impact data [13]. Based on this general framework 14 systematic reviews and maps were planned and started in parallel.
A particular emphasis was placed on an active stakeholder involvement going beyond what is generally done in the course of systematic reviews and maps. A multistep stakeholder engagement approach was developed inspired by two main considerations: (i) the long-standing polarization of the GMO impact debate which is accompanied by a lack of trust among stakeholders and (ii) the concept of Responsible Research and Innovation (RRI) underlying the recent research programmes funded by the European Commission and many national agencies [14][15][16]. The resulting engagement approach, aiming to strengthen the openness, inclusiveness, responsiveness, and transparency of systematic reviews, was implemented for all GRACE reviews and maps in a harmonised way by a team specifically set for the purpose, involving researchers with expertise in natural and social sciences.
According to participant surveys and authors' experiences, stakeholder participation was perceived to be most productive and important in the development and selection of the review questions. As shown in this paper, stakeholders had a considerable impact on the definition and selection of research questions and, thereby, on the scope of synthesis results. These steps are, therefore, the focus of this paper.

Characteristics of the GRACE stakeholder engagement approach for planning the reviews and maps
An overview of the stakeholder engagement approach is provided in Fig. 1.

Openness
Participation in stakeholder workshops was open for all interested stakeholder representatives following a broad invitation circulated to some 500 stakeholder contacts encompassing a broad range of, among others, competent authorities, industry, civil society organisations, professional organisations and researchers. Subsequent steps mainly involved workshop participants.
Despite limited resources, considerable attempts were made to have the key stakeholder groups represented across all steps.

Inclusiveness
As a first step, the overall framework and preliminary review questions were discussed in a 2-day workshop. Stakeholders could provide written comments and questions on the workshop discussions but also on issues not discussed there. Written comments were open to a wide audience including, but not limited to, workshop participants.
The resulting lists of the candidate review questions were prioritized in a second step by stakeholders using an online questionnaire, helping thus the project team identify a final set of review questions.  Fig. 1 The GRACE approach to stakeholder engagement in the planning stage of evidence synthesis. a: Two review protocols on macro level socioeconomic impacts were not published as available journals specialised on publishing such protocols considered them to be out of scope In the third step review protocols were developed and subjected, as drafts, to stakeholder comments in the course of a written consultation. Following a revision of the protocols prompted by stakeholder comments, they were published or prepared for publication as peerreviewed journal papers.
In the fourth step preliminary results of the individual reviews as well as draft general conclusions and recommendations were presented and discussed in the course of another 2 day stakeholder workshop and again subjected to written comments.

Responsiveness
Stakeholders participating in the workshops as well as those provided with consultation materials could provide comments to the review team, which were then collected, categorised and subjected to the individual teams overview through the project' s internal clearing house mechanism. Review teams discussed all comments, revised the review questions and developed written point-by-point responses to all comments and questions.

Transparency
Workshop participants were provided with draft documents and presentations before the workshop meetings. Each step was documented in a detailed stakeholder consultation report which covered workshop inputs and discussions, survey results, written stakeholder questions and comments as well as review team responses. Workshop reports documenting discussions were circulated as drafts to all participants in order to ensure that discussions and viewpoints were accurately depicted. The workshops, the written stakeholder comments, and the review teams' responses are documented in detail in a series of consultations reports available [17][18][19][20][21] on the project website [9].

Reflexivity
To better understand how the GRACE approach was perceived by participants, their views were gathered via phone interviews as well as via online and paper questionnaires.

Status of GRACE systematic reviews and maps
The characteristics described above also accompanied the stakeholder consultations on the preliminary results of the individual reviews and maps (not shown as they are beyond the scope of this paper). Furthermore, GRACE developed general conclusions and recommendations on the use and value of evidence synthesis approaches in GMO impact assessment [22][23][24] which were also included in the scope of the consultations. By the time of the submission of this paper, 7 of the 14 review/map protocols [25][26][27][28][29][30][31] and the results of one systematic review [32] have been published as peer reviewed open-access papers and 10 review/map protocols and the published systematic review are available on the Open Access database CADIMA [33].

Development and prioritization of review questions
The main steps in the development and prioritization of review questions are depicted in Fig. 1-indicating also the methodology used. The manner in which this process influenced the evolution of review questions is indicated in Table 1 and illustrated in Fig. 2.

Proposing and discussing review questions
In the first step a total of 68 preliminary review questions covering the three fields of GMO impact research were subjected to a consultation with 41 stakeholders. During this step the focus was on understanding and discussing the general characteristics of systematic reviews and

Table 1 Development and selection of review questions
Numbers indicate how many review questions entered into the process, were modified and selected. The review questions selected for conducting systematic reviews and maps are detailed in Table 2 a All four review questions were revisited based on stakeholder comments b The prioritization exercise initially produced 11 questions, of which 3 were kept unchanged, 2 were further modified, 6 were dropped and another one added c 27 review questions on micro level socio-economic impacts were combined into one broad systematic map; 4 review questions on macro level socio-economic impacts were selected based on 14 candidate questions-two of the four questions were combined into one review question d 2 questions were dropped as separate questions but included in the scope of other review questions maps, the potential relevance of this method for GMO impact research and assessment, the conceptual frameworks for the review questions, and the preliminary review questions. Based on the workshop discussions and more than 170 written stakeholder comments received after the workshop, the project team modified the review questions considerably in terms of subject, scope, wording and numbers (summarised in Table 1). Of a total of 68 review questions, 23 were revised, 11 dropped and 13 new questions added. For the health and environmental impacts work streams this resulted in a reduction of the number of review questions. For the socio-economic impacts work stream it led to a broadening of the scope and to adding more review questions.

GMO impact area Development and selection of review questions
The resulting 69 candidate review questions were then prioritized by stakeholders in the next step.

Developing criteria and a process for prioritizing review questions
The method and criteria used were tailored to accommodate (i) the specific challenges of dealing with a large number of review questions in three very different scientific and policy contexts in parallel, (ii) the fact that this prioritization was done by stakeholders in a very polarized policy environment, (iii) the constraints of a relatively inflexible research project with tight timelines and resource limitations, and (iv) the exploratory nature of the project in applying evidence synthesis to GMO impact research.
The criteria used were (i) importance for impact assessment, (ii) the existence of expert controversy, and (iii) degree of public awareness (inspired by O'Connner et al. [34] and Clavesi et al. [35]). These three criteria refer to three important dimensions, which also act as three different contexts in the GMO debate: scientific, regulatory, and public dimension. A scoring system (1-5) was used with the option for adding comments allowing for a more nuanced feed-back.

Prioritization of the candidate review questions
The stakeholders participating in the previous consultation step plus those who expressed a particular interest (n = 55) were asked to score each of the 69 candidate review questions in each of the three criteria by using an electronic questionnaire (LimeSurvey).
The respondent rate of the questionnaire was 34.5% (health and environmental impacts) and 29.1% (socio-economic impacts) respectively. Aside from scores, stakeholders provided 74 additional comments. The resulting data set was analysed using SPSS and Excel. Mean scores for each criterion as well as aggregated scores (sum of mean scores of each criterion-assuming an equal relative weight of each criterion) were calculate for each candidate review question and where discussed and considered along with additional stakeholder comments by the three project teams (health, environmental and socioeconomic). An example of how stakeholder scores were reported to the review teams is provided in Fig. 3.
No relative weighting of the different criteria was imposed a-priori, so each review team was essentially free to weight the scores for the individual criteria which allowed considering specifics of their thematic. Moreover, on top of stakeholder priorities and comments, the review teams also needed to consider time, resource, and contractual obligations. All review teams were, however, asked to provide short responses outlining their rationale and how they arrived at the final set of review questions.
As a result, 14 of the 69 proposed candidate review questions were selected and decisions pondered whether they would be pursued as systematic reviews or maps. In the process, and based on the additional stakeholder comments, four review questions on health impacts were revised again, and 27 review questions on micro-level socio-economic impacts were merged into one review question for a broad systematic map (final review questions shown in Table 2).
The systematic approach for processing stakeholder comments described above was particularly important across these steps. It enabled stakeholders and all interested parties to track how their comments and questions were received and processed by the review teams, if comments suggesting modifications were adopted or not, and if not, the reason for which they were not (fully) adopted. Thereby any interested party could track how stakeholders shaped the choice of topics, scope and wording of the review questions.

Experiences and lessons
The previous sections describe the main characteristics of an evidence synthesis engagement approach developed in response to highly polarized views, lack of trust, and the novelty of evidence synthesis for the respective stakeholder community. This section briefly reflects on some of the experiences and lessons learned (also summarised in Table 3). From the viewpoint of the systematic review community, these considerations are limited due to the lack of completed reviews and maps. At this point, therefore, no conclusions can be derived on how the approach described here ultimately affects the relevance and acceptance of the review findings. From the viewpoint of GMO risk regulation, however, it can already at this stage be concluded that the approach describe here appears to be a very interesting and promising alternative option to render GMO risk research more transparent, inclusive and accountable and, that it is definitely worth to be further explored. Fig. 3 Example of scores resulting from stakeholder prioritization. a Review questions 1-4 on Bt crops and Cry toxin. Means of scores for each criterion (importance, expert disagreement, and public awareness) on a scale of 1 (lowest) to 5 (highest). Criteria: importance: the review question is of high importance for the impact assessment of GMOs; expert disagreement: There is expert disagreement on the review question; Public awareness: The review question is the subject of high public awareness. b Review questions 1-4 on Bt crops and Cry toxin 1-4: aggregated mean scores (mean scores across all three criteria). RQ: review questions: RQ1: Does the knowledge about the biology of B. thuringiensis and its action towards organisms (target and non-target) raise any new questions in relation to the risk assessment of CRY toxins produced by GM plants? RQ2: Does the knowledge about the mode of action of CRY toxins at the molecular level pose any issues for the risk assessment of CRY toxins produced by GMPs? RQ3: How is the phylogenetic relationship, related to mode of action and specificity, between CRY toxins? RQ4: What is the current evidence base to conclude on whether the effects of combined, stacked or pyramided CRY toxins, as produced by stacked GMPs, will be additive, synergistic or antagonistic? (Reproduced with permission from [19])

Openness and inclusiveness
The evidence synthesis community frequently highlights the importance of stakeholder involvement in the planning stage of systematic reviews and maps. Review questions should be jointly developed with stakeholders [12] but little reflection is available on what approaches have been used and what have been the experiences. For the GMO impact community, stakeholder involvement in the planning stages of research and assessment is a rare endeavour. Not surprisingly, the GRACE opportunity for upstream involvement was positively perceived by participants with the development and prioritization process of the review questions considered as most relevant step.
Broadly circulated invitations at the beginning of the process (step 1) and the absence of participant selection encouraged a total of 95 stakeholder representatives to participate in one or more steps of the process. It also helped to establish and maintain an atmosphere of openness which is helpful for trust building. The only exception applied to journalists who were excluded from workshops to allow participants to speak freely.
Stakeholder balance was well achieved in all steps with the exception of the written consultation on the review protocols: CSO representatives were provided with the protocol but did not comment. Stakeholder balance across the process is shown in Fig. 4.
Openness measures, however, brought in additional challenges: the consecutive consultation steps in the planning process were conceptualized as a step-by-step learning process, hence, the consultation processes in step 2 (prioritizing review questions) and 3 (on draft review protocols) were limited to those participating in step 1. Still, the number of participants was progressively declining and, more importantly, there was considerable fluctuation among individual participations. Only nine of the 41 stakeholder individuals participating in step 1 continued to step 2.

Table 2 Systematic reviews and maps conducted in the context of the GRACE project
Ref references, SM systematic map, SR systematic review a Protocols for seven systematic reviews or maps resp. and results of one systematic review were published at the time of writing this article. Preliminary results of the systematic reviews and maps are also available from GRACE Stakeholder Consultation Reports and from the final conference [21,24] b Publication of protocols is in preparation c Protocols will not be published as journal papers as they are considered to be out of scope for the main journals publishing evidence synthesis protocols

Health
What are the characteristics of comparative studies of assessment of key chemical crop constituents in GM crops compared to non-GM crops?
What evidence has been collected on the potential toxicity of newly proteins in experimental animals, and what were the characteristics of these studies?
What is the evidence for a changed risk of allergic reactions to an allergenic crop after it has been genetically modified?
What are the characteristics of repeated-dose feeding studies with experimental animals receiving whole food or feed products derived from genetically modified (GM) crops and non-GM counterpart as a control focusing on potential health impacts other than performance?
Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of non-GM maize?
SR [25] What are the effects of the cultivation of GM herbicide tolerant crops on botanical diversity? SR [26] Are population abundances and biomasses of soil invertebrates changed by Bt crops compared with conventional crops?
SR [27] Are soil microbial endpoints changed by Bt crops compared with conventional crops? SR [28] How susceptible are different lepidopteran/coleopteran maize pests to Bt-proteins? SR [29,32] What is the evidence on the inheritance of resistance alleles in populations of lepidopteran/ coleopteran maize pest species?
SM [30] Socio-economic What are the socio-economic impacts of genetically modified crops worldwide? SM [31] What

Recommendations
Openness and inclusiveness Broad stakeholder invitation-any interested stakeholder can participate The absence of participant selection prevented conflicts about the composition of the stakeholder group and contributed to trust building This approach needs supporting measures to ensure that interested stakeholder representatives can participate in all subsequent steps (e.g. travel support, smaller stakeholder groups) Multiple ways of engagement (workshops, written consultations) Online written consultations were mainly used to introduce issues not mentioned in or to clarify or emphasize discussion topics of the face-to-face consultations This set-up is useful for clarifying issues and views and for bringing up issues not addressed during face-to-face meetings, e.g. due to lack of time. It does, however, not help in engaging individuals unable to participate in the face-to-face consultations Familiarizing participants with the evidence synthesis process This measure is very important for a meaningful engagement. This works better if the same representatives of a stakeholder organisation participate in the entire consultation process The engagement process must be tailored to the level of knowledge of and experience with evidence synthesis Measures need to be in place to ensure that the same individuals can contribute over the entire engagement process (e.g. providing travel support and opting for smaller stakeholder groups)

Responsiveness
Procedure for written responses to stakeholder comments The measure was extensively used by stakeholders The procedure resulted in a systematic and transparent consideration of stakeholder comments by review teams and allowed for transparency of how stakeholder inputs shaped the process Time and resource needs are significant and therefore preclude routine application. The procedure appears suitable for sensitive or controversial topics and smaller scale evidence synthesis endeavours Stakeholder can influence topic selection and scope It was challenging to balance stakeholder needs with other project requirements. Many such requirements were contractual (timeline and funding) and therefore given priority Ultimately decisions about modifications of review questions and process need to stay with and be clearly accountable to the review team(s Possible reasons for this participation pattern, as suggested by the authors, are: the high and still increasing number of stakeholder events on GMO issues; the relevance of evidence synthesis for many stakeholders is still not fully clear; the absence of resources to support participation of stakeholders; the lack of familiarity with evidence synthesis (procedure, strengths and weaknesses); the combination of health, environmental and socio-economic topics into the same consultation workshops made it difficult for some people to follow as some topics were outside of their area of expertise (in organisations like competent authorities, industry and research, health, environmental and socio-economic topics are typically covered by different persons); stakeholders felt overloaded with the number of tasks and documents presented to them.
To mitigate such risks, additional measures have to be considered to enable the same individuals to follow the entire process, e.g. smaller groups, more intensive training in evidence synthesis methods and tools, and travel support for stakeholder participants.

Responsiveness
Overall, the GRACE stakeholder consultations led to intense interactions in the course of 2-day workshops and more than 520 written stakeholder comments and team responses. The established procedure to systematically discuss and respond in writing to all stakeholder inputs inspired internal discussions of the review teams. As described in previous sections, stakeholder comments effectively shaped the prioritization, scope and phrasing of the review questions and thereby improved the quality and relevance of the review plans.
On the other hand, stakeholder suggestions sometimes conflicted with resource, time and other requirements of the GRACE project. Occasionally, stakeholder suggestions were contradicting each other, thus it was very important that ultimate decisions remained with the review team.
Review teams, however, needed to be transparent with which inputs they have considered or not, and why. Suggestions not taken sometimes led to disappointment or frustration from stakeholder participants, this being indicative of the need to carefully balance stakeholder expectations and the flexibility to accommodate inputs.

Transparency
Transparency measures were generally appreciated by stakeholders but sometimes resulted in a reluctance of scientists to share preliminary and draft plans with a broader stakeholder group. The reason was the hesitation of some scientists to expose their output to a thorough scrutiny by a polarized stakeholder community at a point in time when such results were considered preliminary or even estimative. Moreover, some scientists pursuing a typical academic career had difficulties to see what they could gain from an 'extended peer review' at this stage.
The procedure for tracing and making transparent how stakeholder inputs were perceived and processed by review teams was-to the best knowledge of the authors-used for the first time in evidence synthesis and was positively received by stakeholder participants and observers [36]. The detailed documentation of all engagement steps also provides a unique evidence base for any ex-post analysis of stakeholder views and their impacts on the project.
The extensive transparency requirements allowed better identification and understanding of divergent views and assessments and were important to establish and maintain the atmosphere of openness despite long standing lack of trust among some stakeholders.
Setting an example for other controversial regulatory science topics?
The consultative approach described here offers interesting advantages if operating in the context of longstanding controversies which often goes together with polarised views and lack of trust. In such contexts the substantial time and resource requirements from stakeholder participants and review teams would also be well justified.
The approach allows to systematically capture and make transparent stakeholder and review team views and priorities. It also provides elements for openly and transparently engaging with stakeholders in planning stages of the review without the ultimate obligation to arrive at a consensus-something which is very difficult to achieve in case of long standing controversies. Still, as shown in this paper, the procedure can help to improve the quality and relevance of review topics and plans.
Another challenge highlighted in this paper is how to operate in contexts with little or no prior experience with evidence synthesis. Although introductions and a training into evidence synthesis were provided this was by no way sufficient. Many stakeholder representatives participating in consultation workshops did not participate in the training and had superficial understandings of the methodology. Moreover, the measures in place to provide openness resulted in participation patterns making stepby-step knowledge generation difficult.
Drawing on this experience the approach could work better in the course of smaller scale endeavours, focussing on a smaller number of review questions in a more homogenous field of research, providing more intense training in systematic review methodology as well as providing resources to stakeholders to encourage continuing participation and tackle attrition.