What is the nature and extent of evidence on methodologies for monitoring and evaluating marine spatial management measures in UK and similar coastal waters? A systematic map protocol

Abstract

Background

Anthropogenic degradation of marine ecosystems is widely accepted as a major social-ecological problem. The growing urgency to better manage marine ecosystems has led to the increasing application of ‘spatial management measures’ including marine protected areas, sectoral (e.g. fishery) closures, and marine spatial planning. However, the designation of varied spatial management regimes is just the first step; achievement of objectives relies upon effective implementation, monitoring, evaluation and adaptation. Despite spatial management being a core component of the marine management portfolio, to our knowledge, there is no systematic overview of the evidence on methodologies available, and employed, to monitor and evaluate their effectiveness across social, economic and ecological outcomes.

Methods

This systematic map will examine existing evidence describing methodologies for monitoring the effects, and evaluating the effectiveness, of marine spatial management across ecological, social and economic outcomes. Our aim is to provide a resource for decision-makers, primarily in the UK but also internationally, that supports effective marine management, and to describe the current evidence base. Identification and evaluation of relevant studies will therefore be restricted to coastal countries identified by our Stakeholder Group as being relevant to the UK, and searches will be restricted to the period 2009 to 2019 to align with the current UK policy context. Searches for relevant grey and academic literature, published in English, will be conducted in four bibliographic search engines, Google Scholar, 38 organisational websites and one specialist data repository. Eligibility screening will be conducted first at title and abstract level, and then at full text. Coding and meta-data extraction from eligible studies will include: bibliographic information, general information about the spatial management measure studied, and methodological information on the monitoring and evaluation undertaken. Consistency checking amongst reviewers will be undertaken during screening, coding and data extraction phases. The outcome of the systematic map will be a database that displays the meta-data of identified relevant studies. Findings will be presented in a descriptive report detailing the evaluation approaches and analytical methodologies employed, and data collection methods applied and/or data required by relevant studies to inform evaluations on the effectiveness of marine spatial management measures.

Background

The world’s marine resources have substantial environmental, social and economic value [1, 2]. Human uses of the seas are diverse, ranging from recreational and tourism activities and cultural heritage, through to more extractive uses such as fishing, dredging, mining, and energy generation. Anthropogenic degradation of marine ecosystems is widely accepted as a major social-ecological problem that could undermine the ability of the oceans to provide fundamental ecosystem services (e.g. [3]). Governments around the world have therefore set out a shared vision to sustainably manage, protect and restore marine ecosystems to achieve healthy and productive seas [4, 5]. These international commitments include a requirement to designate “effectively and equitably managed, ecologically representative and well-connected systems of protected areas and other effective area-based conservation measures… integrated into the wider… seascapes” [4,5,6].

International commitments, together with the growing urgency to protect this value, have led to the increasing application of ‘spatial management’ measures to marine areas (e.g. [7,8,9]). In essence, spatial management aims to incorporate the diversity of human uses, consider the compatibility of different activities, and balance use with the impacts of these activities on biodiversity and people [10]. Spatial management measures typically comprise marine protected areas (MPAs) for biodiversity conservation, sectoral (e.g. fishery) closures to mitigate the effects of their activities and ensure sustainability, and marine spatial plans (‘marine plans’) to integrate social, economic and environmental considerations into proactive management of marine activities. Many countries have already invested substantially in developing an extensive array of marine spatial management measures. For example, the UK currently has 584 MPAs covering approximately 23% of UK domestic waters [7] with 41 more recently designated [11]; has already adopted, or is in the process of developing, a series of regional marine plans for these waters [12]; and has implemented several seasonal fishery closures (e.g. [13]).

Employed effectively, marine spatial management measures can provide a plethora of ecological, social and economic benefits [14,15,16,17], and there has been much work aimed at understanding what effects different spatial management measures have had, to what extent, and the reasons for these outcomes [16, 18, 19]. Such studies can inform the appropriateness of different management options in specific contexts. Yet, initial designation of spatial management is just the first step; achievement of objectives relies upon effective implementation, monitoring, evaluation and adaptation [20, 21]. Effective monitoring is fundamental to document the status of the environment and the activities that occur within it, which in turn informs both the assessment of impacts, including attribution and/or contribution, and the effectiveness of management. Understanding effectiveness then allows for appropriate adaptation of management measures and policy development. However, despite spatial management being a core component of the marine management portfolio, the combined multifaceted complexity of the marine environment, human uses and resultant impacts, makes monitoring and assessing the effectiveness of spatial management an ongoing challenge [20].

With the increased application of marine spatial management measures, governments around the world are now asking how they can effectively and efficiently monitor the marine environment to assess the impacts of spatial management measures (e.g. [22, 23]). Deciding what to monitor, how, and how often, is not straightforward, and the choice of approach can have major implications for costs, efficacy, replicability, and robustness to challenge. Furthermore, the choice of what to monitor and how needs to be informed by a defined evaluation process; however, there is considerable uncertainty as to what evaluation can, and should be, undertaken. Compounding these challenges is the need to improve understanding as to how seasonality can be captured within monitoring and evaluation programs and how to assess the benefits of real-time closures [22]. The vast array of published literature, coupled with the time and resource limitations facing government organisations and agencies, means that maintaining an up-to-date and comprehensive handle on monitoring and evaluation options is unfeasible. Thus, understanding what methodologies are available, and how they are being applied, to monitor and evaluate spatial management effectiveness is critical to ensure cost-effective management and identify priorities for future research to inform and improve management.

To inform this evidence need, we will conduct a systematic map on how the effects of marine spatial management can be determined and what evaluations of effectiveness are undertaken across social, ecological and economic outcomes in coastal countries identified by the Stakeholder Group as being relevant to the UK (see section “Searching for articles” and Table 2 for more information). Systematic maps are typically conducted to map interventions. However, we propose to apply the process in a novel way by focusing on evaluation approaches and analytical methodologies employed, as well as data collection methods applied and/or data required by relevant studies to inform evaluations of the effectiveness of marine spatial management measures across ecological, social and economic outcomes. We will collate, describe, and map the available evidence from existing commercially-published and grey literature, to explore what evaluation and analytical approaches and data collection methods are available, which methodologies are used in different contexts, and whether any are more commonly applied. This study therefore builds on previous systematic maps and reviews on the effectiveness of marine protected areas [19], protected areas more broadly [24], and systematic conservation planning [18] by collating evidence related specifically to monitoring and evaluation. In doing so, we will seek to develop understanding of the methodologies, rather than assess them. By explicitly exploring the methodology behind existing studies, which aim to document effects and effectiveness, we will provide a resource to decision-makers that will help inform discussions regarding the design of appropriate methodologies to incorporate into future monitoring and evaluation plans for marine spatial management.

Stakeholder engagement

The topic and question for this systematic map were originally proposed by the Review Team and co-developed with our Stakeholder Group composed of key stakeholders from UK institutions involved in the monitoring and management of the marine environment, including: Marine Scotland Science (MSS), Natural Resources Wales (NRW), Department of Environment, Agriculture and Rural Affairs (DEARA, Northern Ireland), Inshore Fisheries and Conservation Authorities (IFCAs), Department of Environment, Food and Rural Affairs (Defra), Centre for Environment, Fisheries and Aquaculture Science (Cefas), Joint Nature Conservation Committee (JNCC), Scottish Natural Heritage (SNH) and Natural England (NE). The Stakeholder Group has a diverse breadth of expertise, covering the array of disciplines that will be needed for this systematic map, and extensive experience regarding evidence gaps facing UK and devolved governments. Involvement of a broad group of stakeholders ensures that diverse perspectives are represented and that the resultant map will be of use to policymakers. Discussions were held remotely with stakeholders during protocol development and a face-to-face workshop was held at the University of Salford on 22nd February 2019 with representatives from almost all of the above stakeholder organisations and the Review Team. These engagement activities were designed to formulate and agree the primary and secondary review questions, search strategy, eligibility criteria, and meta-data to be recorded. While the Stakeholder Group will not be involved in the conduct of the review, a second workshop will be held upon completion to disseminate the findings to our Stakeholder Group and identify appropriate pathways and mechanisms through which findings can be disseminated more broadly.

Objective of the map

The primary research question for this map is: What is the nature and extent of evidence on methodologies for monitoring and evaluating marine spatial management measures? This question has the following components:

• Population: areas under marine spatial management in UK and similar coastal waters.

• Intervention: monitoring and evaluation methodologies.

• Comparator: none.

• Outcomes: ecological, social and/or economic outcome measures of interest.

This review will identify and collate retrospective studies that monitor the effects and evaluate the effectiveness of marine spatial management measures across ecological, social and economic outcomes. By ‘monitor’ we refer to techniques applied to observe and measure changes to the state of the marine environment and surrounding communities and industries over time. Techniques reported as being used to monitor a site without any evaluation being undertaken are considered to be ‘monitoring programmes’ and will be excluded from this systematic map. Monitoring is considered to underpin evaluation. By ‘evaluation’ we refer to methodologies for collating and analysing data to determine the effects (the change arising from an intervention) or effectiveness (the degree to which something is successful in producing a desired result) of an intervention against its objectives and/or the resources. We define ‘evaluation’ according to three types, ‘principal’, ‘causative’ and ‘benefit’ which are based on the breadth of evaluation undertaken (Table 1, Fig. 1). Articles will be coded to these categories according to the full breadth of evaluation undertaken, e.g. an article categorised as a ‘causative evaluation’ is also likely to include a ‘principal evaluation’.

Table 1 Typology and definitions of evaluation approaches

Fig. 1

Typology of evaluation approaches. Articles will be classified according to the greatest breadth of evaluation undertaken

We define spatial management as:

• marine protected areas—“a clearly defined geographical space, recognised, dedicated, and managed […] to achieve the long-term conservation of nature with associated ecosystem services and cultural values” [25];

• fishery closures—an area within which fishing by one or more methods, or for particular species, is prohibited on a permanent, seasonal or real-time basis for the purpose of delivering fishery benefits [26]; and

• marine spatial planning (‘marine plan’)—an integrated multi-sectoral plan that informs the current and future distribution of activities in space to maintain delivery of ecosystem services in a way that meets ecological, economic and social objectives [27].

The evidence base will be categorised using a data coding framework (Additional file 1) designed to explore the following secondary questions:

• What approaches and analytical methodologies have been used to evaluate the ecological, social and economic effectiveness of spatial management measures? What data sources do these rely on? What data collection methods are used to gather these? What types of outcomes are measured?

• What techniques exist for understanding the effects/effectiveness of spatial management measures as networks as well as individual sites?

• What monitoring and evaluation techniques are being applied by coastal countries to assess spatial management?

Methods

The systematic map has been developed in accordance with the RepOrting standards for Systematic Evidence Syntheses (ROSES) for systematic map protocols [28] (Additional file 2) and the Collaboration for Environmental Evidence Guidelines and Standards for Evidence Synthesis [29].

Searching for articles

Search string

The following search string will be used:

(marine OR maritime OR coast* OR ocean OR offshore OR inshore OR intertidal OR subtidal OR estuar* OR lagoon* OR fisher*) AND (“protected area” OR “marine reserve*” OR “no-take” OR “no take” OR “marine park” OR “marine sanctuary” OR “ocean sanctuary” OR MPA OR “Natura 2000” OR “special area of conservation” OR SAC OR “site of special scientific interest” OR SSSI OR “special protection area*” OR SPA OR Ramsar OR “marine conservation zone*” OR MCZ OR “marine national monument” OR “closed area” OR (closure* NEAR/10 fish*) OR “fishery exclusion zone*” OR “replenishment zone*” OR “marine plan” OR “marine planning” OR “marine spatial plan*” OR MSP OR “maritime spatial plan*” OR “ocean zoning” OR “spatial management”) AND (monitor* OR evaluat* OR *effect* OR manag* OR impact* OR assess* OR response* OR trend* OR survey*).

A scoping exercise in Web of Science was used to test the specificity and sensitivity of alternate terms, wildcards and Boolean operators (Additional file 3).

Bibliographic databases

We will search for evidence in the following bibliographic databases:

• 1. Web of Science Core Collections.

• 2. Scopus.

• 3. Aquatic Sciences and Fisheries Abstracts.

• 4. Directory of Open Access Journals.

Searches will be performed across all databases using the predefined search string (see Additional file 4 for details of the search string and their adaptation to each bibliographic database). Only articles published in English will be considered; however, all returned non-English articles that pass title-abstract screening will be retained for potential use in future studies (provided the titles and abstracts are also available in English). Time and geographical restrictions (i.e. countries of interest) for the map were agreed with the Stakeholder Group at the workshop held at the University of Salford in February 2019. Searches will be restricted to articles published between 2009 and 2019 and to coastal countries identified by the Stakeholder Group as being relevant to the UK (Table 2). These restrictions were placed to increase relevance to the UK policy landscape [30], to reflect the recent increase in application of marine spatial management measures (particularly MPAs and marine plans) [8, 9], and because we are primarily interested in evaluation approaches and analytical methodologies, and methods applied to collect data to inform these, that are currently being used, or are emerging, that may apply in a UK context.

Table 2 Countries of interest for the systematic map

Search engines

The academic search engine Google Scholar (http://scholar.google.co.uk) will be used to complement searches in traditional bibliographic databases and increase the comprehensiveness of the overall search, particularly for grey literature [31]. Searches will be performed in English using simplified terms from the search string (see Additional file 4). As with searches in bibliographic databases, Google Scholar searches will be restricted to articles published between 2009 and 2019. The first 200 search results will be extracted as citations, following recommendations by Haddaway et al. [31], and added to records from bibliographic databases prior to duplicate removal.

Organisational websites

Searches will be performed across 38 relevant organisational websites and one data repository to capture grey literature using simple search terms related to population and intervention terms. The selected organisational websites were chosen based on their relevance to countries of interest (Table 2), the availability of resources in English, and through discussions with the Review Team and Stakeholder Group. For each website, we will screen the first 100 search results from each search string in situ. Relevant full texts will be recorded for inclusion in the systematic map database. The following information from each search will be recorded and described in the systematic map report: website, date searched, search string used, number of relevant articles identified at full text.

The following organisational websites will be queried:

• 1. Department for the Environment, Food and Rural Affairs.

• 2. Marine Scotland.

• 3. Department of Agriculture, Environment and Rural Affairs.

• 4. Natural Resources Wales.

• 5. Environment Agency.

• 6. Natural England.

• 7. Joint Nature Conservation Committee.

• 8. Marine Management Organisation.

• 9. Northern Ireland Environment Agency.

• 10. Scottish Natural Heritage.

• 11. Scottish Environmental Protection Agency.

• 12. Inshore Fisheries and Conservation Authorities.

• 13. Centre for Environment, Fisheries and Aquaculture Science (CEFAS).

• 14. Seafish.

• 15. Institute for European Environmental Policy (IEEP).

• 16. European Environment Agency.

• 17. European Commission Joint Research Centre.

• 18. HELCOM.

• 19. OSPAR.

• 20. International Council for the Exploration of the Sea (ICES).

• 21. North Pacific Marine Science Organisation (PICES).

• 22. National Oceanic and Atmospheric Administration.

• 23. United States Environmental Protection Agency.

• 24. United States Fish and Wildlife Service.

• 25. Fisheries and Oceans Canada.

• 26. The Australian Institute of Marine Science (AIMS).

• 27. Commonwealth Scientific and Industrial Research Organisation (CSIRO).

• 28. WWF.

• 29. The Nature Conservancy.

• 30. The Organisation for Economic Co-operation and Development (OECD).

• 31. United Nations Environment Programme (UNEP).

• 32. United Nations Environment Programme-World Conservation Monitoring Centre (UNEP-WCMC).

• 33. www.protectedplanet.net.

• 34. OCTO/Open Channels.

• 35. Marineplanning.org.

• 36. IOC-UNESCO marine spatial planning programme.

• 37. International Union for Conservation of Nature (IUCN).

• 38. Convention on Biological Diversity.

A grey literature repository of European government literature, Open Grey (http://www.opengrey.eu/) will also be searched for the period 2009 and 2019.

Supplementary searches

To improve the comprehensiveness of the search, bibliographic searches of all identified relevant review articles will also be undertaken and screened for relevant studies at title, abstract and full text. Articles provided directly by stakeholders will be screened separately to assess relevance.

Estimating the comprehensiveness of the search

Scoping search results (Additional file 3: Table S2) were compared against a test library of 15 publications of known relevance to the review to test and refine the comprehensiveness of the search (see Additional file 3: Table S3 for full list of benchmark articles). These articles were selected by the Review Team, with feedback from the Stakeholder Group, to represent ecological, social and/or economic evaluations of each of the three spatial management types (MPAs, fishery closures, marine plans) considered in this review. The comprehensiveness of the search strategy was assessed by determining the number of benchmark articles that were found during the various iterations of the search string. During the scoping exercise (completed 1st April 2019) 8123 articles were returned using the final search string in Web of Science Core Collection. All 15 articles were located (Additional file 3). Any updates to the search string and search strategy during the conduct of the review will be recorded and all amendments will be reported in the publication of the full systematic map.

Article screening and study eligibility criteria

Screening process

Articles retrieved from bibliographic databases and Google Scholar will be combined into a single library using a review management software (e.g. Endnote). Duplicates will be removed prior to screening.

Articles will be assessed for inclusion according to a hierarchical assessment of relevance: screening article titles and abstracts concurrently, followed by the full text of potentially relevant articles. Where the relevance of articles is unclear at title and abstract stages they will be included and assessed during the full text review. Reasons for exclusion at full text together with details of articles that cannot be located or accessed will be reported in the final review.

Retrieved literature from organisational websites and supplementary searches will be screened separately and articles deemed relevant at full text will be combined with other records prior to compilation of the systematic map.

To ensure consistency at each stage of screening (title and abstract, and full text), a random subset of 10% of articles will be independently screened by all reviewers and Cohen’s kappa statistic [32] will be used to measure the level of agreement between reviewers. The minimum acceptable kappa statistic threshold will be taken as 0.6, which is typically taken to indicate substantial agreement [33]. Nonetheless, given that this threshold is arbitrary, the Review Team will discuss all disagreements irrespective of the score achieved to improve understanding of inclusion criteria and further consistency checking will be conducted on an additional set of articles until kappa scores are above the acceptable threshold.

Demonstrating procedural independence

Members of the Review Team that have authored, or co-authored, articles identified as potentially relevant will refer these to another reviewer for assessment and will not comment on their inclusion or exclusion in the review.

Eligibility criteria

Articles will be screened according to the following criteria:

Relevant population(s): Areas under implemented marine spatial management (fishery closures, MPAs, marine plans) restricted to the identified geographical locations (Table 2). Proposed spatial management measures will not be considered. Large areas (regions, provinces or exclusive economic zones) where broader legislation protects certain species will be excluded from the definition of MPA or fishery closure. Studies with their primary focus on freshwater and/or terrestrial environments will be excluded. Studies will be restricted to those published between 2009 and 2019.

Relevant intervention(s): Monitoring and evaluation methodologies employed to assess effectiveness (Table 1, Fig. 1). Articles that are assessed as being ‘monitoring programmes’ will be excluded.

Relevant comparator interventions: None. Studies will not be required to have a comparator intervention for inclusion.

Relevant study designs: Ecological studies will be required to contain multiple reference sites or a time-series of data to warrant inclusion in the systematic map. Social and economic studies will not be required to have a specific study design. Elements relating to study design (e.g. time-series of data and details of reference sites) will, however, be recorded across ecological, social and economic studies to enable further understanding of evaluation methodologies across different fields of study. Theoretical studies (including predictive modelling studies) and commentary articles will be excluded.

Relevant outcome(s): Any ecological, social and/or economic outcome(s) reported by studies. Broad outcome measures of interest have been defined (see Additional file 1) however outcomes identified within the relevant literature that do not fit within those already defined will be iteratively catalogued into distinct categories. As the focus of the systematic map is on outcomes, studies related to governance or designation process (e.g. administrative, political, legal, planning or design activities) will be excluded. Studies focusing on environmental parameters (e.g. water quality, sediment, etc.) are excluded from the definition of ecological outcomes.

Our aim is to provide a resource for decision-makers, while describing the evidence base. Therefore, we will include both primary (i.e. generation of new data from either field or existing data) and secondary (i.e. literature that consists of analytical interpretations and evaluations that are derived from primary source literature) studies, however these will be documented and reported separately. Studies which report large-scale regional or global evaluations of relevant spatial management measures, that include countries of interest, will also be included in the systematic map. Tertiary literature (i.e. evidence reviews that consist of a distillation and collection of primary and secondary sources but contain no new analysis) will be recorded separately.

Study validity assessment

The validity of articles will not be assessed as part of this systematic map. Elements of study design that might relate to validity (e.g. presence of a reference site, evaluation data timeframe) will be coded as detailed under ‘Data coding strategy’ and Additional file 1).

Data coding strategy

Meta-data, information describing each study, will be extracted from each article considered relevant at full text and recorded using a standardised coding tool (Additional file 1). All coding will be documented in a systematic map database, with each line representing one study outcome measure of interest (i.e. each independent outcome measure considered by each study). Multiple studies reported within one article will therefore be entered as independent lines in the database. Distinct primary articles that report the same study outcome measure of interest based on the same dataset (including those where the dataset has been expanded) as a study published in an earlier article will be linked in the database. During the protocol the 15 benchmark articles were used by the Review Team to refine and reduce ambiguity to pilot the data coding framework (Additional file 1).

The following main categories of data will be extracted:

• Bibliographic information.

• General information about the spatial management measure studied.

• Monitoring and evaluation methodologies.

For the full systematic map, meta-data extraction will be performed by multiple reviewers. Before full data coding commences, consistency checking will be undertaken for coding of a subset of at least 100 studies. All disagreements will be discussed, and coding categories refined if necessary, prior to coding the remaining full texts. Following this, any uncertainties and issues that arise during the data extraction process will be flagged by the reviewer; these will be discussed and resolved by the Review Team in regular meetings. As outlined above, reviewers that have authored, or co-authored, included articles will refer them to an alternative reviewer for coding. If resources allow we may contact corresponding authors by email with requests for missing information or clarifications.

Study mapping and presentation

Results will be published in an open-access academic article in Environmental Evidence journal with a searchable spreadsheet of studies and related coding results.

The systematic mapping process will be represented through a flow diagram describing the number of articles returned by searches, included and excluded during screening stages, and the number of studies included in the final systematic map. Results will be summarised visually and with descriptive statistics including, as a minimum, number of studies by publication year, geographical location, population, intervention, techniques employed, and type of data collected. Results will be categorised by evaluation typology (principal evaluation, causative evaluation or benefit evaluation—see Table 1 and Fig. 1) and focus [ecological, social, economic, socio-economic outcomes or multiple (a combination of these)]. Primary and secondary studies will be reported separately. Studies containing global or large-scale regional evaluations will also be reported separately given that they are likely to contain data from geographical regions beyond those included in this systematic map. Tertiary literature will be listed separately.

Knowledge gaps and clusters will be identified by heatmaps created by cross-tabulating key variables. A second meeting is planned with the Stakeholder Group once data coding is complete to discuss findings and presentation of results. Arbitrary, but specific, cut off points will be agreed at this meeting to identify boundaries (number of studies) at which a topic will be considered as either lacking evidence and therefore being poorly studied, or as having sufficient studies to allow for more meaningful exploration of the monitoring and evaluation methodologies they employ.