What approaches exist to evaluate the effectiveness of UK-relevant natural flood management measures? A systematic map protocol

Abstract

Background

Natural flood management (NFM) measures seek to protect, enhance, emulate, or restore the natural function of rivers as part of approaches to flood risk management (FRM). While there is agreement in both academic and practice/policy literature that NFM, in a general sense, should be part of a holistic FRM strategy to address current and future flood risk, the specifics of how to consistently implement NFM successfully in practice are less well known. This is particularly acute for practitioners in the UK given the nature of the UK’s biophysical and socio-political context. There is a recognition that existing reviews of NFM effectiveness in the UK tend to focus on the natural science basis and it is unclear how studies account for climate change. Further, reviews tend to focus only on UK studies. This systematic map aims to highlight the way in which existing NFM studies, from different disciplinary backgrounds and across Europe, evaluate effectiveness, and the extent to which they account for climate change. This knowledge can help to make recommendations for future areas of research where the multiple issues around understanding effectiveness can be synthesised, and where climate change is systematically taken into account.

Methods

This systematic map protocol addresses the following question: what approaches exist to evaluate the effectiveness of UK-relevant natural flood management measures? The protocol details the methodology that will be used to conduct a systematic map of the range of peer-reviewed journal papers, policy documents, guidance, and other forms of grey literature which currently exist on NFM to give an overview on the way in which the effectiveness of NFM is conceived. The methods detail the search strategy employed for gathering items across the peer-reviewed academic literature and grey literature. Additionally, the methods outline how the reviewers will approach article screening, and the eligibility criteria to include/exclude articles. The methods section also details the steps taken to ensure consistency across all reviewers, the data coding strategy, and methods for presenting the final systematic map. Together, the methods employed will help to identify current knowledge gaps, and will enable recommendations to be made for future research.

Background

Over the past decade, we have seen increased interest in Natural Flood Management (NFM) measures that reduce flood risk through working with, instead of against, the natural environment. These measures include techniques, such as land use management and river restoration, that can be implemented to ‘help to protect, restore and emulate the natural functions of catchments, floodplains, rivers and the coast’ [1]. NFM works well at a catchment level where working with natural environment processes upstream can be used as part of a holistic flood risk management strategy that also encompasses downstream engineered flood defences [2]. Catchment-based flood risk management is a system-based approach that recognises the multiple sources of flood risk, accepting that while single measures to address coastal, pluvial, fluvial and groundwater flooding can work for a period, they neglect the hydrological system in its entirety. For the application of NFM, this conceptualisation necessitates the strategic application of both natural features and measures emulating natural function to support the interception, infiltration and storage of water and the naturalisation of channel flow [3].

European and UK policy has been moving towards the adoption of NFM as part of a holistic flood risk management strategy that can address numerous goals around climate change adaptation, biodiversity and health and well-being. In the UK, the NFM agenda has become more pronounced since 2004 following the Foresight Future Flooding project and the seminal strategy Making Space for Water (2005) which signalled the adoption of a new approach to flood risk management in the UK that sought to work with water rather than against it. In 2017, the UK government allocated a further £15 million of funding to NFM projects and the approach is cemented in the UK Government’s 25-Year Plan (25 YEP) for the Environment which was published in 2018 (see Fig. 1).

Fig. 1

UK policy timeline relating to natural flood management measures

With increasing attention being paid to NFM as an element of flood risk management, there is a need to understand more about the effectiveness of NRM measures, and how effectiveness is being perceived in this context. The promise of NFM lies not only in flood risk reduction but in how NFM measures can help to realise wider ecological, economic and social benefits such as biodiversity restoration, carbon sequestration, improved water quality, and health and well-being [e.g. 3]. Policymakers and other implementers of NFM projects require a sound evidence base to give confidence to their decisions to invest in NFM. Consequently, much research is focussed on gathering underpinning evidence on the effectiveness of NFM. However, the range of NFM benefits means that there are a variety of ways that effectiveness can be framed and measured. Further, considering current effectiveness, climate change means that NFM measures will have to operate in the context of changing climatic conditions which may have a consequential effect on their effectiveness.

There are several existing reviews that directly address or connect to the effectiveness of natural flood management measures, which are based on UK evidence or include evidence that is potentially relevant to the UK. Dadson et al. [4] focus on NFM effectiveness as measured in the natural sciences in terms of reducing flood frequency and hazard. However, NFM can realise multiple benefits beyond flood risk reduction. A wider view of these benefits was taken through an ecosystem adaptation perspective examined in a review by Iacob et al. [5]; however, the review only included a sample of 25 catchment-based projects from the UK, Europe and New Zealand rather than a systematic evidence review. The Environment Agency’s Working with Natural Processes project sought to synthesise UK NFM practice and evidence [6]. The literature review covered evidence from both the academic and grey literature; both nationally and internationally with evidence on the effectiveness of measures identified from the existing knowledge of those involved in the project. Burgess-Gamble et al. [6] mention studies that take account of climate change but future climate was not systematically considered in this review. Further, the focus was largely on UK-only NFM measures with a good range of case studies that try to look at the multiple benefits of NFM as well as providing insights into implementation issues. Given that NFM is also widely adopted in countries with a similar climate and regulatory background to the UK, such as the Netherlands, there is value in assessing evidence from other UK-relevant countries.

There is, consequently, a burgeoning evidence base on NFM and examples of synthesis of that evidence base. However, existing reviews often consider effectiveness a particular disciplinary perspective, e.g. the natural science base [4]. As noted above, NFM benefits, and goals of implementation, may be much wider than simply realising stated outcomes around flood risk reduction. Wingfield et al. [3] point out that: ‘research and resources should be expanded beyond a principles, evidence and efficacy debate to mechanisms of NFM delivery’. This necessitates social scientific analysis and evidence around the framing of policy and, in particular, the processes, opportunities and barriers around NFM implementation in order to develop more politically effective responses to flooding. Both the policy and academic communities recognise that many policy issues in the environmental arena require multiple perspectives that are interdisciplinary and/or cover both the natural and social sciences [7]. Further, there has been a move towards large multi-disciplinary projects within the NFM agenda and, indeed, environmental management more broadly [8]. However, it is not clear how the issue of measuring the effectiveness of NFM is being approached, and indeed how NFM effectiveness is being conceived, particularly when different disciplines are included in evidence reviews.

Several other issues must also be considered when assessing the effectiveness of NFM, whether that be in terms of evidence on outcomes or processes. Firstly, there is an issue around causality or the ‘attribution gap’ [9]. In many cases, a given catchment will need a variety of measures to effectively respond to the risk of flooding, and so it is difficult to isolate the effect of one NFM measure. In addition, the magnitude of a flood will influence the effectiveness of a given NFM measure [4]. Secondly, many studies are based on a short period of observation. However, the effectiveness of NFM may only be apparent over the long term—there may be a significant time lag between implementation and measurable impacts occurring, particularly if the objectives are around achieving multiple benefits such as improved health and well-being [10]. Thirdly, the context and scale of implementation matter when considering NFM, and other nature-based solutions more broadly [3, 11]. Different environmental, economic, social and regulatory factors impact upon the implementation of an NFM particularly because they may be influenced by a wider range of variables than engineered structural defences. Moreover, whilst catchment scale is where NFM should ideally be conceived and implemented, much of the evidence on NFM effectiveness occurs at local scales only [4, 12]. Similarly, the current political institutional context within England and Wales means that development and implementation of NFM is often localised [3]. Finally, there are particular issues to acknowledge when assessing NFM effectiveness taking account of future climate change. Climate data is modelled and so the perceived effectiveness of measures under future climate change will be projected rather than observed. It is not possible to be confident about the climatic and socio-economic conditions under which adaptation measures will operate, as these factors will evolve over time [13]. There may be some inconsistency in the findings of effectiveness studies which take account of climatic change. Moreover, the uncertainty may impact on the performance of measures in practice, which may be better or worse than anticipated.

Acknowledging these challenges, rather than trying to evaluate the effectiveness of NFM by comparing studies between one another (and trying to understand levels of confidence in effectiveness), we will look at how existing studies approach and measure effectiveness following an approach detailed in the ‘Objective of the Review’ section. The systematic map will look for the approach taken to measuring effectiveness, and whether related issues are taken into account such as the scale at which a study takes place, the disciplinary focus of the study, the timeline for assessing effectiveness, and whether effectiveness concerning current and/or future flood risk under a changing climate is considered.

This systematic map will build on previous work but, significantly, departs from that work in three main ways that will complement existing attempts to assess NFM evidence. Firstly, we will explore the approaches taken to assessing NFM effectiveness in order to move beyond the evidence and efficacy debate highlighted by Wingfield et al. [3]. Secondly, the focus on mapping approaches to NFM evidence means that we will consider published works from a variety of different disciplines and, indeed, interdisciplinary work that may approach evidence gathering and effectiveness in different ways. Lastly, we will broaden the time and spatial horizon of the mapping by considering the extent to which climate change is taken account of when assessing evidence and will focus on the inclusion of sources with a similar regulatory and climatic context to the UK (see the ‘eligibility criteria’ section for a detailed explanation). In doing so, we will provide a systematic map that will inform planning and policy-making around discussions related to the framing and measurement of the effectiveness of NFM. We also aim to utilise the review to address issues such as how climate change, and its accompanying uncertainty, may be addressed in future studies on NFM effectiveness within different disciplines.

Stakeholder engagement

Discussions over the formulation of the question took place with members of the advisory group for the Natural Environment Research Council (NERC) Environmental Evidence for the Future initiative, further input from representatives of the UK Environment Agency and the Department for Environment, Food and Rural Affairs (Defra). In discussion with members of the Collaboration for Environmental Evidence (CEE), for example, the decision was taken to remove ‘comparator’ as a review question element. The main research team also assembled an advisory group comprising of academics from disciplines including physical geography, ecology, and planning and environmental management who helped to shape the review question, search strategy, and coding focus.

Objective of the review

The main research question for this evidence map is: what approaches exist to evaluate the effectiveness of UK-relevant natural flood management measures?

The review question has the following key elements:

• Population: Areas in the UK, or areas relevant to the UK, that are susceptible to current and/or future flood risk.

• Intervention: Specific, single NFM measures (listed in Table 1).

  Table 1 Categories of measures within NFM approaches [1]

• Comparator: No comparator necessary.

• Outcome: Impact of NFM measure on current and/or future flood risk, Impact of biophysical, social, and/or political conditions on NFM and vice versa.

The PICO elements of the review question contain no comparator because the review question aims to systematically map how the existing literature approaches the assessment of NFM effectiveness rather than seeking to understand the effect of different NFM measures against, for example, no intervention or an alternative intervention. The review will cover peer-reviewed and grey literature of studies that have examined the effectiveness of NFM measures dependent on a range of outcomes.

We will follow the Environment Agency definition of natural flood management which is ‘implementing measures that help to protect, restore and emulate the natural functions of catchments, floodplains, rivers and the coast’ [1]. The measures included in this definition are shown in Table 1, and it should be noted that this definition excludes measures such as Sustainable Urban Drainage Systems (SuDS) and other urban green infrastructure that seeks to slow down infiltration rates, such as street trees. Definitions of SuDS, such as permeable paving, tend to be hard physical measures that do not ‘emulate the natural functions of catchments, floodplains, rivers and the coast’. We recognise that these terms can be difficult to characterise and the differences between the categories may be contested. The EA’s approach to Woodland Management, for example, captures the scale and location of different woodland types.

We also acknowledge that there are debates over which measures constitute NFM. Wingfield et al. [3] argue convincingly that distinctions hinge on one’s understanding of the word ‘natural’. If naturality is extended to the function of applied measures (as in the case of some sustainable urban drainage technologies such as infiltration trenches) then NFM can include those measures which are applied in order to work with hydrological processes. Within a catchment, the implementation of these emulative measures can, in the event of a flood, divert water away from settlements and significant infrastructure and/or store it to be recycled and re-used as part of other social and/or ecological processes. Conversely, if naturality is taken as referring to those features which occur organically which also help retain, slow the flow of, and/or divert water then NFM is limited to a smaller sub-set of measures. This ambiguity around the word ‘natural’ has led to a series of quasi-synonymous NFM terms such as ‘working with natural processes’, ‘nature-based solutions’ and ‘building with nature’ [14] which, to a greater or lesser degree, relate to the application of technology and may include a wider subset of measures than those shown in Table 1.

All types of flooding (fluvial, pluvial, coastal, groundwater) are included in the systematic map to cover the full gamut of NFM initiatives. The systematic map addresses issues including the different forms of effectiveness of NFM and the extent to which climate change and future flood risk is accounted for. The systematic map will consider NFM in both rural and urban contexts.

We broadly interpret ‘effectiveness’. While ostensibly NFM should achieve its stated aim of reducing flood risk in terms of water volumes and the speed at which water reaches receptors (and even this is beset by difficulties [see: 4]) the fundamental organisational, institutional, and political changes it requires to be successful in this respect also demand recognition as being key to effective measures [15]. If flood risk management is seen as an ongoing, dynamic part of a wider interdependent socio-ecological system which is worked with rather than against [3] then the effectiveness of NFM measures, and evidence of their utilisation, should address the processes, knowledges, actors and norms by which FRM as a whole is practiced as well as flood risk reduction [8, 16].

Drawing on the impact assessment (IA) literature, we identified different forms of effectiveness to provide a basis for conceptualising the multiple factors impacting on the effectiveness of NFM [17]. Within IA, the ‘effectiveness’ forms relate to the focus of measurement where there is; (a) adherence to standardised processes (procedural effectiveness), (b) a contribution to a clearly defined, development-specific goal (substantive effectiveness), (c) time and cost savings that exceed those related to the application of IA (transactive effectiveness), and (d) some form of contribution to broader ideals such as sustainability (normative effectiveness) [18]. Building from this, and seeking to contribute to the literature on understanding NFM effectiveness, we have adopted the ‘forms’ of effectiveness used in the IA literature. Such a move helps to address the observation that NFM not only seeks to reduce flood risk through lowering the volume of water reaching a receptor (or ‘substantive’ effectiveness) but demands new forms of practices, governance and decision-making (i.e. procedural effectiveness). In addition, NFM must be justified on the basis of time and cost efficiency (i.e. transactive effectiveness) as well as, in some cases, contributing to more sustainable forms of flood risk management (i.e. normative effectiveness). This can be seen in Table 2 and diagrammatically in Fig. 2. To this end, we have incorporated effectiveness into our coding strategy based how a paper assesses effectiveness in terms of the four effectiveness forms (see Additional file 5). We have included the coding of climate change as part of normative effectiveness in the extent to which a study addresses both climate change mitigation and adaptation.

Table 2 Identified forms of effectiveness, their measurement and relevance to NFM.

Fig. 2

Model showing the identified forms of approaching effectiveness, and key considerations, to be used as a basis for coding articles

It is also necessary to identify the definitions that will be used to identify the disciplinary background of each study. In terms of the natural sciences, we will include ‘disciplines that deal only with natural events (i.e. independent and dependent variables in nature) using scientific methods’ [19]. We take a broad understanding of the social sciences which includes the study of society and the manner in which people behave and influence the world around them as this helps to avoid questions around human behaviour being framed through a natural scientific lens [20]. We will include issues around the acceptability of NFM amongst particular populations as well as sources that examine issues around governance and knowledge production. We will also follow a more general interpretation of interdisciplinarity which will refer to ‘any form of dialog or interaction between two or more disciplines’ [21].

Methods

The protocol has been developed in accordance with the RepOrting standards for Systematic Evidence Syntheses (ROSES) for systematic map protocols [22] (Additional file 4). In addition, the protocol followed the Collaboration for Environmental Evidence Guidelines and Standards for Evidence Synthesis [23].

Searching for articles

Bibliographic database searches

We will search bibliographic databases using a tested and iteratively modified search string. This will be adapted according to each database’s input syntax. The Boolean version of the search string, as formatted for Web of Science Core Collections, can be found in Additional file 1.

We will search across 13 bibliographic databases as shown in Table 3. Bibliographic database searches will be performed in English only, since these databases catalogue research using English titles and abstracts. A University of Manchester library subscription was used to access bibliographic databases.

Table 3 List of bibliographic databases to be searched for evidence along with the platform and subscription

Web-based search engines

Searches for academic and organisational grey literature will be performed using Google Scholar (https://scholar.google.com/) and searching targeted organisational websites. Haddaway et al. [24] have shown the effectiveness of Google scholar in retrieving grey literature. The Google Scholar search strategy to be employed is shown in Additional file 2. In accordance with best practice guidance, the first 1000 results from each search will be added to the bibliographic database of search results prior to duplicate removal [24]. The web-based search approach will be iteratively updated where necessary according to discussions with the review team and Working and Advisory Groups.

Organisational websites

In addition to Google Scholar, we will search the 21 organisational websites listed in Table 4. For each website, the search terms used in the Google Scholar searches will be applied but will be adapted iteratively in accordance with the search results. Additionally, we will only record items that are available through a free subscription as some organisations charge for certain publications. We will save up to the first 100 search results from each site. These will then be screened for inclusion for analysis in accordance with the study’s eligibility criteria. We will also manually search each website for articles found in publications or bibliography sections. All retrieved sources will be added to the bibliographic database search results prior to duplicate removal. These search activities will be recorded and outlined in a systematic map report.

Table 4 List of organisational websites to be searched for organisational grey literature

Bibliographic searches

Existing reviews on topics of NFM (and related) terms, FRM, and natural resource management more broadly are likely to include articles which have the potential to be within the scope of this study. These will provide a useful way of identifying sources which may have been missed through the various searches. Once screening is complete, manual searches of the bibliographies of previously conducted and relevant reviews will be undertaken. Missed sources will be added to the systematic map database.

Estimating the comprehensiveness of the search

Our test list (Additional file 3) was developed collaboratively with an academic advisory group consisting of environmental and social scientists. It constitutes a list of 18 sources known to be within the scope of the study, including sources that cover a range of disciplinary backgrounds and also included grey literature. The comprehensiveness of the search string was tested by the proportion of these sources that our search string returns. Owing to resource issues, the search string was tested on Web of Science and Scopus only during May 2019. Grey literature on the test list was compared against a list of retrieved sources following a search of 20 organisational websites between 7 and 14 May 2019. Following CEE guidance, we aimed for a 100% return rate [23]. Our assessment of the test list showed that one academic article was not returned by the search string but that material from the same case was present elsewhere in the initial searches (see Additional file 3). Additionally, a further website was added to the list of organisational websites to search for organisational literature.

Search string

The final search string adopted for the search engines (excepting Google Scholar) is as follows (please see Additional file 1).

Flood*

AND

(“natural flood management” OR “nature based solution*” OR “green infrastructure” OR “building with nature” OR “work* with natural processes” OR “ecosystem-based adapt*” OR “sustainable urban drainage system*” OR “sustainable drainage system*” OR “ecosystem service*” OR “river restoration” OR “floodplain restoration” OR “leaky barriers” OR “offline storage area*” OR “catchment woodland*” OR “floodplain woodland*” OR “riparian woodland*” OR “cross-slope woodland*” OR “soil and land management” OR “headwater drainage” OR “run-off pathway” OR “saltmarsh and mudflat*” OR “sand dune*” OR “beach nourishment” OR “manage* retreat*” OR “manage* realignment” OR “enhanced hillslope storage” OR “runoff attenuation” deculvert* OR “slow the flow”).

Please note that the search string above was formatted for use in Web of Science. We also aimed to be inclusive with the search string and included terms such as ‘sustainable drainage system’ since the definitions of various terms, such as sustainable drainage system, nature-based solutions, and green infrastructure, are porous and may overlap with NFM [25]. Included articles that referred to NFM measures not relating to those outlined in Table 1 were removed at the screening stage.

Search update

As the original searches will take place less than a year before the systematic map is produced, no search update will be undertaken as part of this study.

Assembling a library of search results

Following the comprehensive searches, all sources will be gathered and, as an initial step, duplicates will be removed through both automated removal and manual screening in EPPI reviewer.

Article screening and study eligibility criteria

Screening process

Screening will take place sequentially at 3 levels: title, abstract and full text using EPPI reviewer. The process will follow the ROSES Flow diagram (see: Additional file 4).

Members of the research team have worked on and published research in fields related to this study such as: flood risk management; climate change resilience and adaptation; and green infrastructure. These members of the review team, and also the advisory group will be prevented from providing advice or comments relating specifically to studies to which they may have contributed, and will also be prevented from screening or coding any articles to which they have contributed. Any articles authored by a member of the review team will be assigned to a different member of the review team who was not involved in the authorship of the original paper.

Consistency checking

The check for consistency at article screening stage will be undertaken by two reviewers. 10% of all titles and abstracts will be screened and disagreements discussed. Screening criteria will be refined and sharpened in accordance with these discussions and feedback from wider review team. Following Haddaway et al. [26], to test the agreements and disagreements between the two reviewers a Cohen’s Kappa Statistic will be used. This will measure the reliability of agreements between reviewers (i.e. that they are not simply the product of random chance) and will provide a broader measure of reliability of the study’s screening criteria. We aim to have an above 0.60 Kappa rating which denotes ‘moderate agreement’. Where a 0.60 rating is not achieved, a further 10% sub-set of different titles and abstracts will be selected and screened by the same two reviewers. This process will be repeated until a 0.60 rating is achieved. Once 0.60 is achieved, a single reviewer can then go on to review the remaining records.

At full text screening, the consistency check will be undertaken by two reviewers on 5 per cent of articles that are included after title and abstract screening. Where there are any disagreements, the reviewers will meet in person to discuss the approach and reach a consensus. Where consensus is not agreed, then the opinion of a third member of the review team will be sought [23].

Eligibility criteria

This systematic map forms part of a broader research project focused on the usability of evidence for UK practitioners, the mapping protocol required some form of classification for distinguishing between those sources which have relevant lessons for the UK and those which do not.

This study targets sources originating from inside and outside of the UK. However, owing to resources, there was insufficient time to include all NFM literature sources globally, and a research justification oriented around focusing the scope of the study on sources that are UK-relevant has therefore been developed. In order to operationalise this, exclusionary criteria were needed to refine the source list from countries outside of the UK in order to make the systematic map a manageable undertaking considering the time and resources available. The key issue here was to ensure a focus on UK-relevant studies so that the systematic map produces an output that is of value to UK planners and decision makers engaged in NFM activity, thereby meeting the central objective of this NERC funded project. Climate-relevance is used within this study as a key criterion for including and excluding sources of evidence from beyond the UK. Here, the Koppen classification [27] is used to identify UK-relevant sources based on the climate zone that they fall within [28]. Within this study, the focus is on European Union countries and regions falling within the UK’s Koppen Classification type (defined as warm temperate, fully humid, warm summer). This is to ensure that not only do the sources come from locations that share the UK’s climate type, but that they are also covered by a similar regulatory regime concerning legislation such as the European Floods Directive. Other countries and regions from outside of the European Union also fall into this classification type including New Zealand, South Eastern Australia, Serbia, South West Chile and the Pacific coast of North America, and literature sources from these countries are consequently excluded.

Any approach to excluding literature sources addressing NFM measures will have its deficiencies, and climate-relevance is no exception. Indeed, the climate within Northern and Western Europe, which contains most of the UK’s climate-relevant countries and regions, varies significantly at finer levels of granularity. Climate was considered alongside a range of other types of exclusionary criteria with our advisory group of researchers specialising in topics linked to NFM, and the decision was taken to focus on climate as a key exclusionary criteria. Here, the perspective is taken that sources of NFM evidence originating from European Union countries and regions falling within the UK’s Koppen Classification type are of particular relevance to the UK because of their climate similarity and the resulting influence of the climate on flood risk and NFM. The decision was taken to focus on sources of literature emerging from the UK and countries within the European Union that share the UK’s current climate type, according to the Koppen Climate Classification, as opposed to future analogues. Future climate change is uncertain and multiple different future climate scenarios are therefore produced [29]. We do not know with any certainty which countries represent realistic future climate analogues for the UK, and this approach was therefore avoided.

Regulatory relevance was also identified as being important and, for this reason, we focussed only on European sources. European countries follow the Water Framework Directive (2003) and the Floods Directive (2007). The Floods Directive was introduced in order to ensure better internal and external coordination between member states and policy fields, as well as to manage upstream and downstream flood risks [30]. Whilst there are important differences between the integration of the floods directive within national contexts, due to the principle of ‘transposition’, the floods directive ‘reinforced the need to consider future risks such as climate change and urbanization’ particularly through flood risk mapping and the flood risk assessment process [30]. Given the widespread implementation of the floods directive across Europe, only European countries with similar climates to the UK are included in the analysis. Conversely, given that countries such as the US and China, who both conduct much research on NFM, have different regulatory and climatic contexts, these studies were excluded from the analysis. Similarly, whilst Australia and New Zealand may have similar climatic contexts, in part, the regulatory contexts are significantly different and therefore both countries were excluded from the analysis. European countries without a similar climatic context to the UK (e.g. Italy, Greece, and so on) were discounted even though the regulatory context around the Floods Directive is largely the same.

In terms of the screening process, it is anticipated that at the title and abstract level, the eligible population in terms of UK-relevant climatic and regulatory contexts will be readily identifiable. At least initially, the title and abstract should also give a good indication as to the eligibility of the intervention, although it is recognised by the research team that an abstract may, for example, make general allusions to sustainable urban drainage systems (SuDS) but that these may include evaluations of specific NFM measures (e.g. soil management) within the full text. At the full-text stage it is anticipated that the more specific aspects of the population and intervention criteria will be identifiable, as will criteria relating to the comparator and outcome. It is recognised that the criteria for eligible outcomes is broad—this was a deliberate strategy to ensure a comprehensive range of potentially eligible sources were retrieved across a range of different disciplines.

The specificities of the population and outcomes will be addressed and elucidated through our coding strategy. This is because it is unrealistic to expect definitive yes/no answers regarding a source’s analysis of, for example, the conditions which contribute to flood risk as this can be an issue of interpretation and can therefore give rise to potential bias in selection. As previously noted, there are also inherent difficulties in isolating and measuring the effectiveness of NFM as a result of the complex systems within which they are implemented combined with the uncertainties associated with climate projections if future flood risk is explored. Therefore, using the coding strategy to grade and categorise a source’s attempts at addressing the conditions and effectiveness of NFM was considered a more useful approach. The originality of an article, where originality refers to the provision of empirical evidence, will be used to exclude sources that state the benefits of NFM without providing original empirical evidence that contributes to the NFM evidence base.

Eligible population:

Study addresses current and/or future flood risk, and is relevant to the UK (see Table 5).

Table 5 Countries/regions similar to the UK climate based on the Koppen Climate Classification and shared regulatory context.

Eligible intervention:

Study analyses an aspect of the effectiveness of specific, single NFM measures, NFM measures studied have been knowingly and deliberately implemented to manage flood risk.

Eligible comparator:

N/A.

Eligible outcome:

Study produces an original evaluation/analysis of the effectiveness of NFM on current and/or future flood risk.

Study evaluates the impact of NFM on biophysical, political, and/or social conditions and/or vice versa.

Criteria applied at title and abstract stage:

• Exclude sources which do not cover a specific NFM measure (or measures) listed in Table 1.

Criteria applied at full-text stage:

• Exclude sources which are historical in their evaluation of NFM (e.g. addresses historic inundation levels using paleoecological techniques).

• Exclude sources which amalgamate the impacts of several NFM measures (e.g. at a catchment scale) into an overall assessment of flood risk and therefore do not provide an evaluation of single NFM measures.

• Exclude sources which re-state the benefits of NFM with no original research.

• Exclude sources which have no applicability to the UK by way of clear differences in climate and regulatory context.

A full list of sources excluded at full-text screening stage will be included in the resultant report of the systematic map with reasons given for the exclusions.

Full text retrieval

We will attempt to retrieve full texts of relevant abstracts using University of Manchester library subscriptions. Where full texts cannot be retrieved, we will make use of institutional access provided to our Advisory Team members. If records still cannot be obtained, requests for articles will be sent to corresponding authors where emails are provided and/or requests for full texts will be made through ResearchGate.

Study validity assessment

This systematic map will not assess study validity, which follows the guidance for systematic maps by CEE [23].

Data coding strategy

Data will be extracted and coded according to a range of variables listed in Table 6 (see Additional file 5 for the full data coding strategy). As the key points of interest for the study centre are around the forms of effectiveness of NFM studies, data coding has been developed to include different forms of effectiveness as well as factors that impact upon effectiveness such as country of study, scale of analysis, and time period covered by the analysis. Owing to time and resource restrictions, we will not contact authors to clarify any missing or unclear information. In these cases, we will clearly indicate missing data.

Table 6 Categories of data analysed

To ensure consistency at full text screening stage, 10 to 15 full texts will be coded by two reviewers with discussion over disagreements following the initial coding. Where disagreements cannot be resolved, we will use a third member of the team to try to resolve the disagreement as this strategy has been used in previous reviews [e.g. 31]. Alterations to data categories, variables, or their descriptions may be made following the resolution of disagreements and all screening disagreements and any alterations will be recorded in order to ensure replicability.

Study mapping and presentation

The presentation of the systematic mapping results will be discussed with our advisory group. Our initial intention is to display the systematic mapping results using the ROSES flow diagram (see Additional file 4). The results will be presented using descriptive plots and tables showing the numbers of studies across the variables listed in. Within each data category heat maps will be used to illustrate the abundance of the existing evidence in relation to the variables within each category.

Knowledge gap and cluster identification strategy

We will aim to use illustrations (e.g. pivot charts) to display the volume of evidence across multiple dimensions of data category by offsetting variables from two or more data categories against one another. This will be used to identify knowledge gaps (i.e. where links between variables are under-represented by existing evidence) and knowledge clusters (i.e. where data on links between variables is over-represented relative to other links). Examples of variables that will be used together include (this should be read as an indicative rather than exhaustive list):

• Type of NFM measure versus time-scale of effectiveness.

• Type of NFM measure versus temporal distribution.

• Result of NFM measure versus policy sphere.

• Scientific discipline versus penetration.

• Type of NFM measure versus climate change considerations.