Why is the evidence base for effectiveness of win–win interventions to benefit humans and biodiversity so poor?
© Pullin. 2015
Received: 31 August 2015
Accepted: 4 September 2015
Published: 22 September 2015
Within global efforts to improve human wellbeing and conserve biodiversity there is increasing interest in resolving conflicts between these goals and achieving win–win outcomes. In fact, these win–win outcomes are viewed as fundamental to achieving sustainability goals of the Convention on Biological Diversity.
Consequently a variety of actors, intergovernmental, governmental, non-governmental and private are funding and/or implementing interventions (projects) in the field with the twin objectives of improving human wellbeing and conserving biodiversity. A range of interventions has been implemented, such as community-based management of natural resources, payment for ecosystem services and alternative livelihoods. Some projects have now been running for decades whilst others are very recent. All are aimed at influencing human behaviour and enabling community change that both improves human health, income, social capital etc and impacts positively on biodiversity either directly (e.g. through creating incentive to conserve to provide income) or indirectly (e.g. through decreasing pressure on natural resources). Although these actors can generally point to one or two examples where their interventions appear to work, there is an alarming lack of systematic evaluation and synthesis of evidence for their effectiveness. Given the level of investment and the importance of achieving these goals it is interesting to ask why the accumulation of rigorous evidence is so poor.
Intergovernmental agencies of the World Bank and United Nations which are in turn funded by National Governments (tax payers). Often their funding is matched by private sources.
International non-governmental organisations such as conservation or international development not-for-profit organisations (NGOs). Their funding in turn comes from members and donors but also from 1 to 3.
Private donors (philanthropists) who may fund directly or through 1 and 2 above
National Government agencies using tax payers money.
Who implements these projects is even more complex and includes International NGO’s (2 above) or national NGO’s, often in partnership with National or Regional Government agencies. Big projects may involve multiple funding and implementing organisations.
The amount of funding involved is difficult to estimate with any accuracy but it is safe to say that it is billions of dollars per annum. Much of this is tax payer’s money and so it is legitimate to ask about the return we are getting on our investment, how the implementers are performing, which interventions are working and why and how we can learn from previous projects and invest limited resources more wisely. These would seem reasonable questions for any investors or donors to ask. The questions do not challenge the objectives of these projects, they challenge the processes used to achieve them. It would be unreasonable to expect all projects to work optimally from the start but it seems that we are not systematically gathering the evidence required to help us understand what works and what doesn’t, where and when, in order that we might learn how to optimise their effectiveness.
The standard way of asking these questions of individual projects is through the process of monitoring and evaluation. If good evaluation study designs are implemented from the beginning of the project, indicators adequately monitored, data analysed and transparently reported, then there is much to learn from the results. Donors might expect that this was routine for projects involving such heavy investment and globally important objectives. So where are these evaluations that answer the above questions and what do they tell us?
Considerable effort has now been expended by the Collaboration for Environmental Evidence (CEE) community on a range of systematic reviews (SRs) looking at evidence of effectiveness and impact of some of these types of intervention. All of the CEE SRs completed to date have concluded that sufficient evidence to inform future decisions is lacking.
For example, a review of the effectiveness of community forest management concluded that the evidence base was poor and evidence of effectiveness equivocal . The authors subsequently suggested a suite of recommendations to improve the evidence base .
A review of the impacts of terrestrial protected areas (PAs) on human wellbeing found little evidence of win–win outcomes and a generally poor evidence base insufficient to help predict positive or negative outcomes from PA establishment .
Two recent related systematic reviews on the effects of decentralised forest management  and payment for ecosystem services  reported limited evidence as the major outcome, particularly in relation to human wellbeing indicators.
The findings of these SRs suggest that evidence most often comes from multiple small scale, short term, opportunistic studies (often PhD studies) that are poorly resourced. These studies often have limitations in their design due to resource constraints and timescale issues. They often commence after the implementation of the project and lack baseline measurements. Comparators are often unsuitable and limit ability to attribute change to the intervention. Large scale, long term, robustly designed evaluations are lacking or at least not readily available to independent reviewers.
With such important global goals and such large scale investment in interventions why is the evidence base so poor? Firstly I should clarify that the ‘evidence base’ I refer to is a shared and open body of evidence with transparent reporting of data. Organisations may perform internal evaluations but unless these are openly reported and made freely available they do not contribute to the global evidence base. Consequently, it is not the total existing evidence that counts but only the available evidence. Non-reporting or private reporting of evaluations is therefore one potential cause of the paucity of evidence.
At the global level conservation and development is a competitive business. Organisations compete for funding and for influence at national and regional levels. Winning funding involves maintaining a high profile and portfolio of success. Attracting members and donations is best done by focussing on actions and successful outcomes rather than objective reporting of all outcomes. The incentive to report failure is absent and in such a competitive climate it is only by making monitoring and evaluation a mandatory element of the funding package that the evidence will be made available. Winning funding as an organisation should also involve demonstrating cost effectiveness but often the cost element of a project is paramount and the drive is for cost-minimisation. Minimising expense on monitoring and evaluation is one way of reducing the costs of the project; a short term gain for the implementing organisation (they get the funding) but a long-term loss for the global community (we don’t get the evidence to help us learn and become more effective). The motivations for the implementing organisations in a competitive world are easy to understand but the motives of the funders are less clear. Lack of rigorous monitoring and evaluation is allowing an ineffective and scattergun style funding landscape where allocation of large-scale funds is continuing in the absence of evidence for effectiveness as a criterion for project funding.
As one might expect, the headline risk is that billions of dollars are being wasted on ineffective and even counterproductive projects. Certainly, the available evidence does not refute this assertion. On a more operational level, mixed results of projects may be discouraging implementing organisations from open reporting and so also preventing learning/sharing of what works and what doesn’t. This creates a kind of vicious circle of confidentiality and/or ignorance that needs to be broken to enable collection and consideration of evidence that can be used to improve outcomes of future projects and prevent using ineffective interventions over and over again.
CEE SRs provide the ‘evidence for lack of evidence’ and challenge the funding and implementing organisations to provide better and more transparent monitoring and evaluation reports. The funders must drive this change by making robust monitoring and evaluations a mandatory element of projects. Although this has been called for repeatedly over many years it is only now, with the advent of CEE SRs, that we can see the reality of lack of evidence against a smokescreen of selective reporting of success.
One might speculate on how valuable a new intervention might be if it showed the potential for achieving such a fundamental win–win objective as human wellbeing and biodiversity conservation. It is just a short step to think about investment in robust evidence generation and systematic evidence synthesis as the means to identify that intervention. CEE SRs will continue to examine and report the evidence base for the effectiveness of important global interventions that affect our environment. My hope is that funders will use the outcomes of CEE SRs to influence implementing organisations to improve the standard of design and reporting of project evaluations and therefore to improve our evidence base, project effectiveness and the return on society’s investment with which they are entrusted.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Bowler D, Buyung-Ali L, Healey JR, Jones JPG, Knight T, Pullin AS. The evidence base for community forest management as a mechanism for supplying global environmental benefits and improving local welfare. CEE review 08-011 (SR48). 2010. www.environmentalevidence.org/completed-reviews/the-evidence-base-for-community-forest-management-as-a-mechanism-for-supplying-global-environmental-benefits-and-improving-local-welfare.
- Bowler DE, Buyung-Ali LM, Healey JR, Jones JPG, Knight TM, Pullin AS. Does community forest management provide global environmental benefit and improve local welfare? Front Ecol Environ. 2012;10:29–36.View ArticleGoogle Scholar
- Pullin AS, Bangpan M, Dalrymple SE, Dickson K, Haddaway NR, Healey JR, Hauari H, Hockley N, Jones JPG, Knight TM, Vigurs C, Oliver S. Human well-being impacts of terrestrial protected areas. Environ Evid. 2013;2:19.View ArticleGoogle Scholar
- Samii C, Lisiecki M, Kulkarni P, Paler L, Chavis L. Effects of decentralized forest management (DFM) on deforestation and poverty in low and middle income countries: a systematic review. CEE 13-015a. Collaboration for Environmental Evidence. 2014a. www.environmentalevidence.org/completed-reviews/effects-of-decentralized-forest-management-dfm-on-deforestation-and-poverty-in-low-and-middle-income-countries.
- Samii C, Lisiecki M, Kulkarni P, Paler L, Chavis L. Effects of payment for environmental services (PES) on deforestation and poverty in low and middle income countries: a systematic review. CEE 13-015b. Collaboration for Environmental Evidence. 2014b. www.environmentalevidence.org/completed-reviews/effects-of-payment-for-environmental-services-pes-on-deforestation-and-poverty-in-low-and-middle-income-countries.