Inclusive development and prioritization of review questions in a highly controversial field of regulatory science

Abstract

How to best assess potential health, environmental and other impacts of genetically modified organisms (GMOs) and how to interpret the resulting evidence base have been long-standing controversial issues in the EU. As a response, transparency and inclusiveness became a major focus of regulatory science activities in the GMO impact area. Nevertheless, nearly three decades of controversies resulted in a heavily polarized policy environment, calling for further efforts. Against this backdrop the EU funded project GRACE explored the value of evidence synthesis approaches for GMO impact assessment and developed an evidence synthesis framework with a strong emphasis on openness, stakeholder engagement, transparency, and responsiveness to tackle regulatory science challenges. This framework was tested and implemented in the course of 14 systematic reviews or maps conducted on selected review questions spanning potential health, environmental, and socioeconomic impacts of GMOs. An inclusive development and prioritisation of review questions is of key importance in evidence synthesis as it helps to provide a better link between stakeholder demands and concerns and policy relevant outcomes. This paper, therefore, places a particular focus on the stakeholder involvement strategy developed and experiences gathered during this particular step in the course of the GRACE project. Based on this experience, possible lessons for future engagement exercises in highly controversial fields of regulatory science are discussed.

Background

Evaluating potential health, environmental and socioeconomic impacts of genetically modified (GM) crops has been a dynamic field of research since the early 1990s. Interpreting the resulting primary data has frequently been a subject of controversial debates augmented by the complexity and the diversity of test designs and the multitude of endpoints under investigation [1] as well as by contradicting results. Reviewing secondary data by academia and regulatory committees has frequently led to discussions about, among other, studies not (appropriately) considered, relative weight attached to data generated by different methods, bias, lack of transparency and to divergent conclusions [2,3,4,5,6,7,8].

Against this backdrop the EU funded research project GRACE (GMO Risk Assessment and Communication of Evidence) set out to explore the use of evidence synthesis approaches to assess and synthesize exiting evidence on potential health, environmental and socio-economic impacts of GM crops [9]. At the time of the planning and start of this project (2011–2012) systematic reviews and maps had sparingly been applied to GMO impact research.

GRACE had a twofold aim: (i) to explore the suitability of systematic reviews and maps in GMO impact research and assessment, and (ii) to assess the available evidence on selected research questions. The results were expected to support evidence-based policy making in GMO impact assessment.

By adapting existing guidance documents [10,11,12] GRACE developed a general framework appropriate for the synthesis of GM crop impact data [13]. Based on this general framework 14 systematic reviews and maps were planned and started in parallel.

A particular emphasis was placed on an active stakeholder involvement going beyond what is generally done in the course of systematic reviews and maps. A multi-step stakeholder engagement approach was developed inspired by two main considerations: (i) the long-standing polarization of the GMO impact debate which is accompanied by a lack of trust among stakeholders and (ii) the concept of Responsible Research and Innovation (RRI) underlying the recent research programmes funded by the European Commission and many national agencies [14,15,16]. The resulting engagement approach, aiming to strengthen the openness, inclusiveness, responsiveness, and transparency of systematic reviews, was implemented for all GRACE reviews and maps in a harmonised way by a team specifically set for the purpose, involving researchers with expertise in natural and social sciences.

According to participant surveys and authors’ experiences, stakeholder participation was perceived to be most productive and important in the development and selection of the review questions. As shown in this paper, stakeholders had a considerable impact on the definition and selection of research questions and, thereby, on the scope of synthesis results. These steps are, therefore, the focus of this paper.

Characteristics of the GRACE stakeholder engagement approach for planning the reviews and maps

An overview of the stakeholder engagement approach is provided in Fig. 1.

Fig. 1

The GRACE approach to stakeholder engagement in the planning stage of evidence synthesis. a: Two review protocols on macro level socio-economic impacts were not published as available journals specialised on publishing such protocols considered them to be out of scope

Openness

Participation in stakeholder workshops was open for all interested stakeholder representatives following a broad invitation circulated to some 500 stakeholder contacts encompassing a broad range of, among others, competent authorities, industry, civil society organisations, professional organisations and researchers. Subsequent steps mainly involved workshop participants.

Despite limited resources, considerable attempts were made to have the key stakeholder groups represented across all steps.

Inclusiveness

As a first step, the overall framework and preliminary review questions were discussed in a 2-day workshop. Stakeholders could provide written comments and questions on the workshop discussions but also on issues not discussed there. Written comments were open to a wide audience including, but not limited to, workshop participants.

The resulting lists of the candidate review questions were prioritized in a second step by stakeholders using an online questionnaire, helping thus the project team identify a final set of review questions.

In the third step review protocols were developed and subjected, as drafts, to stakeholder comments in the course of a written consultation. Following a revision of the protocols prompted by stakeholder comments, they were published or prepared for publication as peer-reviewed journal papers.

In the fourth step preliminary results of the individual reviews as well as draft general conclusions and recommendations were presented and discussed in the course of another 2 day stakeholder workshop and again subjected to written comments.

Responsiveness

Stakeholders participating in the workshops as well as those provided with consultation materials could provide comments to the review team, which were then collected, categorised and subjected to the individual teams overview through the project’ s internal clearing house mechanism. Review teams discussed all comments, revised the review questions and developed written point-by-point responses to all comments and questions.

Transparency

Workshop participants were provided with draft documents and presentations before the workshop meetings. Each step was documented in a detailed stakeholder consultation report which covered workshop inputs and discussions, survey results, written stakeholder questions and comments as well as review team responses. Workshop reports documenting discussions were circulated as drafts to all participants in order to ensure that discussions and viewpoints were accurately depicted. The workshops, the written stakeholder comments, and the review teams’ responses are documented in detail in a series of consultations reports available [17,18,19,20,21] on the project website [9].

Reflexivity

To better understand how the GRACE approach was perceived by participants, their views were gathered via phone interviews as well as via online and paper questionnaires.

Status of GRACE systematic reviews and maps

The characteristics described above also accompanied the stakeholder consultations on the preliminary results of the individual reviews and maps (not shown as they are beyond the scope of this paper). Furthermore, GRACE developed general conclusions and recommendations on the use and value of evidence synthesis approaches in GMO impact assessment [22,23,24] which were also included in the scope of the consultations. By the time of the submission of this paper, 7 of the 14 review/map protocols [25,26,27,28,29,30,31] and the results of one systematic review [32] have been published as peer reviewed open-access papers and 10 review/map protocols and the published systematic review are available on the Open Access database CADIMA [33].

Development and prioritization of review questions

The main steps in the development and prioritization of review questions are depicted in Fig. 1—indicating also the methodology used. The manner in which this process influenced the evolution of review questions is indicated in Table 1 and illustrated in Fig. 2.

Table 1 Development and selection of review questions

Fig. 2

(The bar chart was reproduced with permission from [19])

Example how stakeholder priorities and comments influenced type, scope and wording of review questions (described in detail for all review questions in [17,18,19,20])

Proposing and discussing review questions

In the first step a total of 68 preliminary review questions covering the three fields of GMO impact research were subjected to a consultation with 41 stakeholders. During this step the focus was on understanding and discussing the general characteristics of systematic reviews and maps, the potential relevance of this method for GMO impact research and assessment, the conceptual frameworks for the review questions, and the preliminary review questions.

Based on the workshop discussions and more than 170 written stakeholder comments received after the workshop, the project team modified the review questions considerably in terms of subject, scope, wording and numbers (summarised in Table 1). Of a total of 68 review questions, 23 were revised, 11 dropped and 13 new questions added. For the health and environmental impacts work streams this resulted in a reduction of the number of review questions. For the socio-economic impacts work stream it led to a broadening of the scope and to adding more review questions.

The resulting 69 candidate review questions were then prioritized by stakeholders in the next step.

Developing criteria and a process for prioritizing review questions

The method and criteria used were tailored to accommodate (i) the specific challenges of dealing with a large number of review questions in three very different scientific and policy contexts in parallel, (ii) the fact that this prioritization was done by stakeholders in a very polarized policy environment, (iii) the constraints of a relatively inflexible research project with tight timelines and resource limitations, and (iv) the exploratory nature of the project in applying evidence synthesis to GMO impact research.

The criteria used were (i) importance for impact assessment, (ii) the existence of expert controversy, and (iii) degree of public awareness (inspired by O’Connner et al. [34] and Clavesi et al. [35]). These three criteria refer to three important dimensions, which also act as three different contexts in the GMO debate: scientific, regulatory, and public dimension. A scoring system (1–5) was used with the option for adding comments allowing for a more nuanced feed-back.

Prioritization of the candidate review questions

The stakeholders participating in the previous consultation step plus those who expressed a particular interest (n = 55) were asked to score each of the 69 candidate review questions in each of the three criteria by using an electronic questionnaire (LimeSurvey).

The respondent rate of the questionnaire was 34.5% (health and environmental impacts) and 29.1% (socio-economic impacts) respectively. Aside from scores, stakeholders provided 74 additional comments. The resulting data set was analysed using SPSS and Excel. Mean scores for each criterion as well as aggregated scores (sum of mean scores of each criterion—assuming an equal relative weight of each criterion) were calculate for each candidate review question and where discussed and considered along with additional stakeholder comments by the three project teams (health, environmental and socioeconomic). An example of how stakeholder scores were reported to the review teams is provided in Fig. 3.

Fig. 3

(Reproduced with permission from [19])

Example of scores resulting from stakeholder prioritization. a Review questions 1–4 on Bt crops and Cry toxin. Means of scores for each criterion (importance, expert disagreement, and public awareness) on a scale of 1 (lowest) to 5 (highest). Criteria: importance: the review question is of high importance for the impact assessment of GMOs; expert disagreement: There is expert disagreement on the review question; Public awareness: The review question is the subject of high public awareness. b Review questions 1–4 on Bt crops and Cry toxin 1–4: aggregated mean scores (mean scores across all three criteria). RQ: review questions: RQ1: Does the knowledge about the biology of B. thuringiensis and its action towards organisms (target and non-target) raise any new questions in relation to the risk assessment of CRY toxins produced by GM plants? RQ2: Does the knowledge about the mode of action of CRY toxins at the molecular level pose any issues for the risk assessment of CRY toxins produced by GMPs? RQ3: How is the phylogenetic relationship, related to mode of action and specificity, between CRY toxins? RQ4: What is the current evidence base to conclude on whether the effects of combined, stacked or pyramided CRY toxins, as produced by stacked GMPs, will be additive, synergistic or antagonistic?

No relative weighting of the different criteria was imposed a-priori, so each review team was essentially free to weight the scores for the individual criteria which allowed considering specifics of their thematic. Moreover, on top of stakeholder priorities and comments, the review teams also needed to consider time, resource, and contractual obligations. All review teams were, however, asked to provide short responses outlining their rationale and how they arrived at the final set of review questions.

As a result, 14 of the 69 proposed candidate review questions were selected and decisions pondered whether they would be pursued as systematic reviews or maps. In the process, and based on the additional stakeholder comments, four review questions on health impacts were revised again, and 27 review questions on micro-level socio-economic impacts were merged into one review question for a broad systematic map (final review questions shown in Table 2).

Table 2 Systematic reviews and maps conducted in the context of the GRACE project

The systematic approach for processing stakeholder comments described above was particularly important across these steps. It enabled stakeholders and all interested parties to track how their comments and questions were received and processed by the review teams, if comments suggesting modifications were adopted or not, and if not, the reason for which they were not (fully) adopted. Thereby any interested party could track how stakeholders shaped the choice of topics, scope and wording of the review questions.

Experiences and lessons

The previous sections describe the main characteristics of an evidence synthesis engagement approach developed in response to highly polarized views, lack of trust, and the novelty of evidence synthesis for the respective stakeholder community. This section briefly reflects on some of the experiences and lessons learned (also summarised in Table 3). From the viewpoint of the systematic review community, these considerations are limited due to the lack of completed reviews and maps. At this point, therefore, no conclusions can be derived on how the approach described here ultimately affects the relevance and acceptance of the review findings. From the viewpoint of GMO risk regulation, however, it can already at this stage be concluded that the approach describe here appears to be a very interesting and promising alternative option to render GMO risk research more transparent, inclusive and accountable and, that it is definitely worth to be further explored.

Table 3 Overview of experiences and recommendations for evidence synthesis in controversial contexts

Openness and inclusiveness

The evidence synthesis community frequently highlights the importance of stakeholder involvement in the planning stage of systematic reviews and maps. Review questions should be jointly developed with stakeholders [12] but little reflection is available on what approaches have been used and what have been the experiences. For the GMO impact community, stakeholder involvement in the planning stages of research and assessment is a rare endeavour. Not surprisingly, the GRACE opportunity for upstream involvement was positively perceived by participants with the development and prioritization process of the review questions considered as most relevant step.

Broadly circulated invitations at the beginning of the process (step 1) and the absence of participant selection encouraged a total of 95 stakeholder representatives to participate in one or more steps of the process. It also helped to establish and maintain an atmosphere of openness which is helpful for trust building. The only exception applied to journalists who were excluded from workshops to allow participants to speak freely.

Stakeholder balance was well achieved in all steps with the exception of the written consultation on the review protocols: CSO representatives were provided with the protocol but did not comment. Stakeholder balance across the process is shown in Fig. 4.

Fig. 4

Stakeholder balance for each engagement step. N number of stakeholder participants in each step

Openness measures, however, brought in additional challenges: the consecutive consultation steps in the planning process were conceptualized as a step-by-step learning process, hence, the consultation processes in step 2 (prioritizing review questions) and 3 (on draft review protocols) were limited to those participating in step 1. Still, the number of participants was progressively declining and, more importantly, there was considerable fluctuation among individual participations. Only nine of the 41 stakeholder individuals participating in step 1 continued to step 2.

Possible reasons for this participation pattern, as suggested by the authors, are: the high and still increasing number of stakeholder events on GMO issues; the relevance of evidence synthesis for many stakeholders is still not fully clear; the absence of resources to support participation of stakeholders; the lack of familiarity with evidence synthesis (procedure, strengths and weaknesses); the combination of health, environmental and socio-economic topics into the same consultation workshops made it difficult for some people to follow as some topics were outside of their area of expertise (in organisations like competent authorities, industry and research, health, environmental and socio-economic topics are typically covered by different persons); stakeholders felt overloaded with the number of tasks and documents presented to them.

To mitigate such risks, additional measures have to be considered to enable the same individuals to follow the entire process, e.g. smaller groups, more intensive training in evidence synthesis methods and tools, and travel support for stakeholder participants.

Responsiveness

Overall, the GRACE stakeholder consultations led to intense interactions in the course of 2-day workshops and more than 520 written stakeholder comments and team responses. The established procedure to systematically discuss and respond in writing to all stakeholder inputs inspired internal discussions of the review teams. As described in previous sections, stakeholder comments effectively shaped the prioritization, scope and phrasing of the review questions and thereby improved the quality and relevance of the review plans.

On the other hand, stakeholder suggestions sometimes conflicted with resource, time and other requirements of the GRACE project. Occasionally, stakeholder suggestions were contradicting each other, thus it was very important that ultimate decisions remained with the review team.

Review teams, however, needed to be transparent with which inputs they have considered or not, and why. Suggestions not taken sometimes led to disappointment or frustration from stakeholder participants, this being indicative of the need to carefully balance stakeholder expectations and the flexibility to accommodate inputs.

Transparency

Transparency measures were generally appreciated by stakeholders but sometimes resulted in a reluctance of scientists to share preliminary and draft plans with a broader stakeholder group. The reason was the hesitation of some scientists to expose their output to a thorough scrutiny by a polarized stakeholder community at a point in time when such results were considered preliminary or even estimative. Moreover, some scientists pursuing a typical academic career had difficulties to see what they could gain from an ‘extended peer review’ at this stage.

The procedure for tracing and making transparent how stakeholder inputs were perceived and processed by review teams was—to the best knowledge of the authors—used for the first time in evidence synthesis and was positively received by stakeholder participants and observers [36]. The detailed documentation of all engagement steps also provides a unique evidence base for any ex-post analysis of stakeholder views and their impacts on the project.

The extensive transparency requirements allowed better identification and understanding of divergent views and assessments and were important to establish and maintain the atmosphere of openness despite long standing lack of trust among some stakeholders.

Setting an example for other controversial regulatory science topics?

The consultative approach described here offers interesting advantages if operating in the context of long-standing controversies which often goes together with polarised views and lack of trust. In such contexts the substantial time and resource requirements from stakeholder participants and review teams would also be well justified.

The approach allows to systematically capture and make transparent stakeholder and review team views and priorities. It also provides elements for openly and transparently engaging with stakeholders in planning stages of the review without the ultimate obligation to arrive at a consensus—something which is very difficult to achieve in case of long standing controversies. Still, as shown in this paper, the procedure can help to improve the quality and relevance of review topics and plans.

Another challenge highlighted in this paper is how to operate in contexts with little or no prior experience with evidence synthesis. Although introductions and a training into evidence synthesis were provided this was by no way sufficient. Many stakeholder representatives participating in consultation workshops did not participate in the training and had superficial understandings of the methodology. Moreover, the measures in place to provide openness resulted in participation patterns making step-by-step knowledge generation difficult.

Drawing on this experience the approach could work better in the course of smaller scale endeavours, focussing on a smaller number of review questions in a more homogenous field of research, providing more intense training in systematic review methodology as well as providing resources to stakeholders to encourage continuing participation and tackle attrition.