Science informing Policy – a health warning for the environment

Environmental scientists regularly receive encouragement to engage with both their policy counterparts and wider society, to increase the effectiveness with which they communicate their science and to demonstrate its beneficial impact on wider society [1–3]. A significant body of literature now exists on ways in which environmental scientists might engage with non-scientists and be understood [4] and representative scientific societies and funders of science provide advice on how scientists can increase their impact (e.g. Ecological Society of America, British Ecological Society, UK Natural Environment Research Council) and, in some cases, provide funds and training for such activities. The need for good science to inform environmental policy decisions is arguably more urgent than ever before and it follows that good translation of science both for policy communities and wider society, needs encouragement, but it does also need some methodological basis.

The development of mutual understanding between scientist and policy maker of the role of scientific evidence in informing policy is critical to effective environmental management. A higher profile and greater media coverage such engagement might provide are potentially good things for environmental science and scientists. The value of original research needs greater appreciation by those who might pay for it (tax payers, donors). Truly novel and groundbreaking research should be widely and accurately reported using skilled journalism to explain often complex issues to a lay readership or viewing audience. But, along with these positives there are potential negatives when scientists or vested interest groups compete for attention to promote one piece of research as more important or interesting than another.

Communicating science to a wider audience is already something of an industry. Many publishers now seek to gain media coverage for individual papers and summarize the findings of key papers in dedicated sections of their journals to broaden their impact. Of course, where there is commercial interest there will always be selective relationships between scientists and those that might seek to exploit the results. Many organisations provide a service for those who wish to translate their outputs and increase their impact in a commercial setting (e.g. healthcare [5]). Putting commercial interests to one side, there are many situations in which collective (rather than selective) scientific evidence should be used for the collective good of society; public health and environmental management are perhaps two examples. Unfortunately, in contrast to public health, there is no widely adopted mechanism in environmental management to facilitate this process.

Whilst we recognise that the translation of environmental science into policy is not a simple linear process [6] and that there are many other factors that influence policy decisions, there are also cases where science has been fundamental to the policy change. Although we may all be able to name some good examples (e.g. combating acid rain and protection of the ozone layer), the threat always exists of bad science being more effectively communicated than good [7]. Examples illustrating a number of potential dangers of promoting individual studies or sectoral opinion of science, with the intention of influencing policy, have been recorded in the health sciences. In the health sector, such issues are often acute and have been extensively studied. The following list serves as a ‘health warning’ and is only illustrative and not exhaustive.

The first problem concerns apparently conflicting findings, timescale and scientific progress. Findings of early research in human nutrition led to development of guidance which, as research progressed, was demonstrated to have been too simplistic. For example, early advice for diabetics was to follow a low carbohydrate diet; the reasoning being that carbohydrates are metabolised to glucose which is problematic for diabetics. However, later research demonstrated that such extreme dietary restrictions are not necessary and so the new message is to eat a ‘healthy diet’ and avoid foods containing ‘simple’ sugar – rather than all carbohydrates. To the general public however, this can be interpreted as a U-turn on policy, which can lead to lack of confidence in the ability of scientists to get it right.

Often apparent lack of consensus arises, not because the science changes, but because selective messages are given out by those with vested interests. If such messages are more effectively promoted than the actual science, the general public may find it difficult to differentiate between sources of information; “..but they say now that it isn’t bad for you after all!”. To the recipient, sound scientific advice becomes one and the same as marketing materials and subjective media opinion. In such cases, it can be difficult to pursue a consistent policy in the face of apparently contradictory science.

A different problem arises when high quality science accumulates that challenges current dogma. Despite the weight of evidence shifting there is no direct or timely translation into policy or practice if the translators (individuals or groups) are themselves overly ‘conservative’ or have long held ‘beliefs’ in which they have invested time, effort and reputation. Science often moves faster than an individual’s or group’s opinion of the current evidence. A landmark example of the dangers of overreliance on established expert opinion was provided by Antman et al. [8] who were concerned with the use of thrombolytic therapy (so-called ‘clot-busting’ drugs) in Acute Myocardial Infarction (heart attack) (AMI). They examined “conventional expert wisdom”: analysing ’expert’ review articles and textbook chapters to see what recommendations these made about the use of the therapy in AMI and found that it was not until 1986 that the majority of experts recommended it as a routine treatment. They then searched for all Randomised Control Trials of thrombolytic therapy in AMI, undertook a cumulative meta-analysis of the data and showed the clear benefit of the therapy in reducing death from AMI. Their analysis demonstrated that this benefit became clear around 1975, some 10 years earlier than when use of the therapy in AMI was regarded as ‘routine’ in expert texts. Even then, there were some textbooks/reviews which did not mention thrombolytic therapy at all or were still reporting it as ‘experimental’. During those intervening years, it is quite probable that thousands of people with AMI died, who could have survived, if the evidence had been systematically gathered and synthesised and the findings effectively communicated and applied by ‘experts’.

A third problem is that of ‘myth generation’ where a story is given the status of being science-based, through repeated assertion and dissemination, or where ‘weak’ evidence is acted upon as if it were strong. Myth generation occurs in at least the following two different contexts.

The first case is where a policy is in place that is to some extent based on evidence and a myth is generated based on inferior science that undermines that policy. The so-called ‘MMR controversy’ in the UK is a salutary lesson in the dangers of one study being promoted (in this case by the non-scientific media and some stakeholder groups) without due criticism of its quality and without reference to the cumulative evidence-base; either because the messengers are not equipped with the skills to assess the science and/or because to do so would ruin the story or oppose their beliefs. In 1998, a paper by Wakefield et al. was suggestive of a link between the ‘triple vaccine’ for Measles, Mumps and Rubella (MMR) and the onset of autistic spectrum disorder in children [9]. The paper was widely reported in the British media and sparked a rapidly escalating public debate about the safety of the vaccine. Despite key and high-level statements about the problems with the study and the strength of the evidence-base which demonstrated no such links, uptake of the first dose of the MMR vaccination by age two years in the UK population fell from 92% before the controversy (1995/6), to just under 80% by 2003/04 [10]. The World Health Organisation recommends that immunity levels of 95% are needed to prevent outbreaks of disease [11] and in England and Wales there were 56 confirmed cases of measles in 1998, 438 in 2003 and 1370 in 2008 [12]. The original paper has since been retracted by Wakefield’s co-authors [13]. Efforts to promote the results of single studies without critical appraisal and out of context of the background evidence-base will run the risk of undue attention being given to a fragment of the evidence-base with the possible consequences of a damaging ‘MMR’ type fiasco in environment management.

A second case of myth generation is where there is a need for evidence on which to base policy and some information comes to light to fill the void. The information may be based on poor science but is acted upon as if it were much sounder than it really is. An environmental example comes from Australia, where the cane toad (Bufo marinus) is an invasive alien species that was originally introduced as a potential biological control agent, but today serves as a textbook example of how such plans can go badly wrong. The cane toad is still spreading through Northern Australia threatening many native species, due to its toxicity to almost anything that tries to eat it, and costing many millions of dollars in attempts to control it. No surprise then that any scientific finding that might address this problem would be quickly assimilated by the policy community. In their paper appropriately titled ‘The myth of the toad-eating frog’ Shine et al. [14] describe a circulating story purported to be based on evidence that the tadpoles of the native frog Litoria dahlii was able to consume the tadpoles and eggs of the cane toad without ill effect and were therefore a potential agent with which to combat the invader. Some scientific credibility had been given to this claim by a short article in a scientific journal [15] and the story was subsequently widely reported in the media. However the study design described in this article appears flawed and Shine et al. report on a subsequent more rigorous replicated laboratory trial which suggests that L. dahlii is no different from other native frog species in being susceptible to cane toad toxin and suffering significant mortality when attempting to feed on cane toad tadpoles. This example illustrates how apparently science-based information can be picked up by the media as an interesting story, without any critical appraisal of its validity, and become accepted as evidence on which to base action.

At the core of these examples is the fundamental problem of bias. None of us is free from it and just as there are sound reasons for using experimental methodology to reduce the influence of bias in primary research, we need methodologies that perform the same role in translation of evidence to inform decisions. There are many forms but simple selection bias, where a single study can be actively seized upon and promoted by partisan groups because it strengthens their stance, is an adequate example here. The MMR story is a case of this, where those opposed to the vaccine in principle contributed to the media promotion of Wakefield’s study. The field of nutrition yields examples of the food industry selecting and promoting single studies, regardless of their research quality, which support their marketing claims even though they might be counter to the evidence base as a whole [7]. Members of the public may find it difficult to identify when scientific information is coming from an independent scientific body or from those with a vested interest. Apparently conflicting information may be interpreted as a lack of scientific consensus making it easier for a biased point of view to be promoted by those with clever marketing and big budgets. There is potential advantage in presenting conflicting findings as lack of consensus, particularly if that consensus is likely to be harmful to one’s enterprise. The tobacco industry has been able to use this tactic for many years and the climate change sceptics have waged a similar media offensive. There are many genuine conflicts in environmental management where stakeholder groups seek to use science to press their case (e.g. fisheries management, alternative energy generation, industrial pollution, pest control) and policy must see through any bias and objectively assess the weight of evidence.

Along with providing examples that warn of biased use of science there are useful lessons to be learnt from the experience of the health sector in combating the problem. A shared evidence base formed using a recognised methodology, open not only for information but also for critical appraisal and contribution, has the potential to reduce error and bias, dilute the impact of vested interest and absorb the latest evidence into a synthesis of all available evidence. Evidence synthesis provides a buffer against irresponsible reporting of scientific findings (e.g. through media or vested interest groups). The evidence base acts both as a benchmark of current knowledge and a filter of new findings. Random interjection of evidence without filter is inefficient and can be extremely costly. The evidence base should be formed by openly collaborative independent organisations using systematic review and evidence synthesis methodologies to provide an independent consensus view that is constantly updatable as new science becomes available. Examples are familiar in the health service (e.g. http://www.cochrane.org), but also exist in social justice, education and international development (e.g. http://www.campbellcollaboration.org).

Environmental policy should be informed by an evidence base that the environmental community shares and not be unduly influenced by the latest finding, the largest effect, the most interesting result, or the most contradictory or controversial. The concept is simple; new findings are reported in the context of the existing evidence base and not as isolated, independent fragments of science. An equivalent independent organisation in the environmental sector is now functional ([16]; http://www.environmentalevidence.org) but is yet to be as influential as in other sectors. Form and function are well developed and the key challenge is to grow to the size (the number of systematic reviews of relevant questions and equivalent information) required to be considered mainstream practice. So who contributes and who pays? If the Collaboration for Environmental Evidence is to grow to the required size and level of activity a broad spectrum of environmental scientists needs to become engaged in systematic review and evidence synthesis. Unfortunately incentives are few but there are signs that this is changing. Whilst there is no clear funding stream to develop evidence-based practice in the environmental sector and no career structure within institutions that encourages the development of expertise in systematic review and evidence synthesis, there are early signs of increases in both demand-led activity from commissioners requiring an evidence base and bottom up activity from scientists wanting to test and improve the methodology in their field.

With a collective effort among multiple actors in the environmental sector an effective and rapid expansion of evidence-based practice and evidence-informed decision making can be achieved. We suggest the policy community should encourage development of an independent evidence-base in environmental management by (1) demanding rigorous systematic reviews when commissioning work to address needs for evidence; (2) discouraging ‘special relationships’ between scientists (including representatives of vested-interest groups and subsets of the science community such as government-funded centres) and policy formers and (3) engaging experts in critical appraisal of the evidence and avoiding overreliance on expert opinion to establish the evidence base. We further suggest research funders should encourage the development of evidence synthesis as a specialism and promote evidence synthesis alongside applied primary research programmes. Evidence synthesis serves as a pathway to impact, as an identifier of knowledge gaps and primary research priorities and as an independent assessor of the quality of the science being funded. For example, consider the idea that the policy impact of a paper may be objectively measured by whether or not it has been included in a systematic review commissioned by an appropriate policy body. Additionally, the methodological quality of that paper could be objectively measured by its impact on the systematic review outcome in comparison with other included papers. If policy impact is to become a measure of assessment of individual research outputs then there is potential for the ‘every-researcher for himself’ scenario of evidence dissemination to get worse as researchers are pressurised to provide evidence of impact. This will not help develop objective measures of impact but will increase the problems listed above. Alternatively the promotion of a synthetic approach to policy-relevant science might improve the evidence base and provide a more objective measure of the impact of individual papers.

The value and credibility of science is undermined by examples such as those given above. Translation of science to inform policy should be an ordered process with rules to minimise bias equivalent to those established for primary research. In environmental sciences it seems we are still being encouraged to engage in an entrepreneurial free-for-all effort to promote individual studies to the policy community, a state that the health scientists have fought against for some 20 years. Are we going to learn from their pioneering efforts or perform a costly reinvention of the wheel?