What do we know about the impacts of the Marine Stewardship Council seafood ecolabelling program? A systematic map protocol

Abstract

Background

Voluntary sustainability standards and eco-labels are market-based mechanisms used to encourage producers and consumers toward environmental sustainability. The Marine Stewardship Council (MSC) is one such program which aims to improve the state of the world’s oceans and promote a sustainable seafood market. Now in its 21st year, with approximately 14% of global fisheries landings certified, there is growing evidence of the program’s impacts (direct and indirect) on factors from fisheries management and consumer awareness to coastal communities’ livelihoods and international law. To better understand the program as a whole, the proposed systematic map will collate and describe published research on the environmental, social and economic effects of the MSC program, and indicate the prevalence of disciplines or topics and study designs in this literature. Areas considered of greater interest, knowledge gaps, and future research priorities will be identified.

Methods

This systematic map protocol describes how research regarding the MSC will be searched, identified and described. All research on the MSC and its effects (direct and indirect) will be included. The review is not limited to effects on certified fisheries but will include those on supply chain companies, socio-economics of coastal communities, governments, biological populations and ecosystems, NGOs and other stakeholders impacted by or potentially influenced by the MSC. The search scope includes studies from MSC’s foundation in 1997 to the present. To identify studies, pre-determined inclusion and exclusion criteria will be used at the title, abstract and full text levels. In addition to the use of bibliographic databases and internet search engines, the authors will call for and search for grey-literature. The final systematic map will be presented in a descriptive report detailing the focus, extent, and occurrence of research on the MSC’s impacts, taking special care to map the disciplines focused on the programme and the study design of research.

Background

The Marine Stewardship Council (MSC) is an international voluntary sustainability standard-setter and seafood ecolabelling program. Established in 1997, the MSC’s mission is to contribute to the health of the world’s oceans by promoting sustainable fisheries and effective management through its third-party certification scheme [1]. By utilizing its ecolabel as a market-based incentive and rewarding sustainability, the MSC encourages improvements ‘on the water’, by certifying fisheries that have demonstrated good environmental practices as articulated in the MSC’s Fishery Sustainability Standard [2]. For a product to exhibit the MSC ecolabel, all companies in the supply chain that handled that product must be certified under the MSC Chain of Custody (CoC) Standard [2] which guarantees full traceability of the product from a certified sustainable source.

The MSC Fisheries Standard defines fishery sustainability on the basis of the guidelines of Food and Agriculture Organisation of the United Nations (FAO) code of conduct for responsible fisheries [3], and strives to incorporate internationally-established best practice and best available science. The Fishery Standard requirements focus on three principles: healthy target stock status; mitigation of environmental impacts; and effective management [4]. In order to achieve and/or maintain certification, fisheries might need to, among other things, reduce fishing effort, establish new harvest strategies, mitigate fishing impacts on vulnerable species, habitats and ecosystems, change governance or policy practices, or define fishery-specific management.

MSC’s mission as an organisation is underpinned by its theory of change (ToC), see Fig. 1. A ToC describes how or why a change or desired outcome is hypothesised to occur within a given context. The MSC’s ToC suggests that recognising sustainable fishing practices with certificates and eco-labels creates market incentives to improve fishing practices. When a consumer chooses to purchase MSC certified seafood over another seafood product, certified fisheries are rewarded for their sustainable practice through that market preference. Furthermore, purchasing preferences may go beyond individual consumers to include processors, suppliers and retailers who can make commitments to sourcing MSC certified products. Such preferences may increase the global demand and market access for certified fisheries and provide the incentives needed for fisheries to undergo the rigorous assessments to potentially become MSC certified. Increasing numbers of certified fisheries, coupled with greater consumer awareness may incentivise further fisheries that are operating below the MSC standard to make sustainable changes ‘on the water’ in line with the organisation’s principles. For example, to align with principle 2 of the Fishery Standard (minimising environmental impact) a fishery may adjust its practices to ensure that other species and habitats in the ecosystem remain healthy. This could mean fishing at night to avoid accidental bird entanglements or collecting information on seabed impacts so as to better manage the potential effects of fishing activities.

Fig. 1

The Marine Stewardship Council’s theory of change describes how the organisation envisages itself contributing to more sustainable seafood practices

In order to qualify for MSC certification a minimum score of 60 must be achieved for each of the 31 principle issues in the Standard and fisheries must achieve an average score of 80 overall for each of the three Principles (determined by the average of the principle issue scores under that Principle) [5]. Where a fishery scores above 60 but below 80, a fishery is given a ‘condition’ where that score must be improved to the 80 level over the certification period (usually 5 years) [5]. The MSC allows fisheries to qualify for MSC certification without meeting the 80 level on all indicators because the movement of fisheries from the 60 to 80 levels is where fisheries are able to transparently improve their practice and undergo surveillance in this period. If a fishery fails to a ‘close’ a ‘condition’ within a set time period, their certification will be taken away. This is one of the tenets of MSC’s theory of change as it contributes to improved fisheries practice. This process is broadly outlined in Fig. 1.

This process can result in environmental impacts (changes in the health of target fish stocks or more effectives fisheries management) as well as impacts on the market (greater demand and/or supply of sustainable seafood or consumers willing to pay price premiums for eco-labelled products). Importantly the changes that occur due to fisheries voluntarily complying with the MSC’s standards or through the ‘closing’ of ‘conditions’ directly align with the principles and criteria in the standards themselves. For example, Principle 1 states that: “The fishing activity must be at a level which is sustainable for the fish population. Any certified fishery must operate so that fishing can continue indefinitely and is not overexploiting the resources” [4]. Here, to attain a score of 80 (for one of the principle issues within Principle 1) there “shall be evidence that the stock is at the target reference point now or has fluctuated around the target reference point for the past few years” [4]. As such the term ‘sustainable’ is linked to the target reference point of a stock (i.e. the amount of biomass of a particular stock believed to allow the population to maintain a sustainable yield). Thus, the MSC can only claim to create changes in line with this definition of ‘sustainable’. In other words, the ToC does not just outline potential sustainable changes on the water but specific changes that directly align with the principles and criteria in the MSC’s Standards.

The potential changes described in the ToC and their outcomes have been described in published literature. These include effects of MSC certification on fisheries management [6, 7], supply chains [7, 8] and the eco-labels impact on consumer awareness [9, 10]. There are further published reported impacts that are not described in Fig. 1. These include effects on local economies [11], coastal communities [8], and international law [12]. It is this diverse range of impacts that this systematic map aims to capture. To this end, a comprehensive catalogue of relevant studies across disciplines, methods and geographic regions will provide an overview of knowledge clusters and knowledge gaps. Such an overview will contribute to elucidate the state of knowledge on how fisheries sustainability standards perform and operate, and answer Cooke et al. “Call for evidence-based conservation and management of fisheries and aquatic resources” [13]. This is important given that a search of articles published by Environmental Evidence, the official journal of the Collaboration of Environmental Evidence, finds few systematic maps and reviews that deal explicitly with marine fisheries except Araujo et al. [14], Hughes et al. [15] and Leisher et al. [16] and none that focus on certification and/or eco-labels.

While literature regarding the MSC has been included in previous systematic mapping and review exercises [17,18,19], these studies have focused exclusively on final outcomes of certification and sustainability standards (e.g. practice adoption or environmental impacts). Here, we seek a comprehensive overview of all the aspects of the MSC program that researchers have focused on, including management and policy processes, consumer awareness and willingness to pay, to name a few.

Systematic reviews of different sustainability standards rely on evidence typology and inclusion criteria [e.g., PICO (population, intervention, comparator, outcomes) or study design such as BACI (Before, after, control, intervention)] that may be difficult to apply to studies of changes in fisheries management [20]. For example, wild capture fisheries occur in highly dynamic environments, with harvesters and marine populations shifting spatially and temporally. These are harder to monitor than crops or forests that are spatially fixed and clearly delimited entities. In addition, potentially broad spatial scales over which some species are distributed, (such as the wide migratory range of tuna populations), or the vertical distribution of vulnerable ecosystems in the water column, make it costly and difficult to conduct randomised control trials or collecting information on covariates that control for such effects. This means that very strict selection filters risk excluding a large proportion of studies and, with them, potentially valuable insights. To avoid this, the PICO framework will be applied. Studies that lack comparators will still be included but will be noted separately. Also, any study design will be included.

Herein summary, we propose to map all the literature on the MSC without narrowing the scope based on study design. To our knowledge such a broad yet systematic review exercise has not been undertaken before. This document has been developed following the RepOrting standards for Systematic Evidence Syntheses (ROSES) for systematic map protocols [21] and the CEE Guidelines and Standards for Evidence Synthesis [22].

Stakeholder engagement

There are various definitions of stakeholder in the literature. According to the ‘ROSES for systematic map protocols’ a definition of stakeholder should be used to include researchers and decision-makers with a stake in the systematic map being created. As seen in Table 1 of Haddaway et al. [23] the term can refer to “People that have an interest in the subject matter: includes researchers and experts”, “someone who has a stake in the findings” or “those …that use the information from a systematic review”. Haddaway et al. [23] also encourages a broader inclusion of stakeholders potentially affected by the results of the map. As such, the following methods and questions have been formulated and developed with input from MSC staff and external researchers who have expertise in fisheries biological sciences, marine ecology, the development and application of the MSC’s standards, social science, and economics. The stakeholders were chosen for their expertise, covering the array of disciplines that tend to focus on the MSC, thus ensuring that all aspects and perspectives of the programme will be captured in the mapping process, as well as taking diverse perspectives into account. The MSC staff have a stake in the findings, are most likely to be affected by the review and have decision-making capabilities. The external researchers are interested in the subject matter and may use the information from the review. While this is not an exhaustive representation of all stakeholders potentially affected by the results of the review, these stakeholders will nevertheless “ensure that inputs and outputs are of the greatest relevance and reliability to all interested parties” [23]. The stakeholder described will provide input throughout the review process including the coding, analysis and write-up stages. Meetings with stakeholders were held at the MSC headquarters office in London on 25 April 2018 and remotely on 30 April 2018 to formulate the research questions, search techniques, inclusion and exclusion criteria and discuss the available evidence in general.

Objective of the review

This proposed systematic map aims to identify evidence of the impacts of MSC and to collate and describe the topic areas of research undertaken to date on the environmental, social and economic effects of the MSC program. It will also collect and describe the type of evidence thee articles draw on. This will help identify areas most commonly researched, trends in the research over time, and knowledge and data gaps, and will help identify potential future research priorities for external researchers as well as MSC’s Monitoring and Evaluation program.

Primary question

What is the evidence for impacts of the MSC program?

This question has the following components:

• Population Any harvester groups or chain of custody companies, governments, communities, biological populations and ecosystems potentially impacted by MSC certification or have the potential to become MSC certified and thereby potentially impacted by such certification.

• Intervention MSC certification (according to the MSC Fisheries or Chain of Custody standards).

• Comparator Absence of intervention either between fisheries, countries, sites or groups, and/or over time.

• Outcome Positive, negative, neutral or ambivalent impacts on ecosystems, trade, socio-economic outcomes, governance, management or policy.

Secondary questions

What are the characteristics of documented evidence in terms of geographic location, and focus of the study (e.g. consumer willingness to pay, program legitimacy and credibility, fisheries bycatch etc).

• What are the characteristics of the fisheries that are most reported in the literature?

• What are the primary disciplines that focus on the MSC programme and its effects?

• What study designs are used in research regarding the MSC programme?

• What are the types of outcomes (e.g. socio-economic, ecological, policy recommendations etc) for which evidence is documented?

• What is the frequency of these documented outcomes?

• What are the current knowledge clusters? To be identified by examining the characteristics of the fisheries, disciplines and study designs.

• Where do gaps exist in the evidence base that may be prioritised for future research?

Methods

Searching for articles

The articles to be screened for the systematic map will be found using the following Boolean string:

“Marine Stewardship Council” OR “fisheries certif*” OR “certif* fisheries” OR “seafood eco-label” OR “seafood ecolabel” OR “seafood eco label”.

This search string is applicable to all three of the databases that will be searched (see below). No restrictions on document type will be applied, however only literature published between 1997 and the present will be included as 1997 is the year of MSC’s founding. Note that the first fisheries were certified in 2000.

All database, search engine and grey literature searches will take place in English.

Searches in the publication databases will be performed using university library subscriptions from university-affiliated researchers on our team (FN, AL, HT and GP). Where an article is unavailable using one institution’s subscriptions, access to it will be sought elsewhere by another team member.

Estimating the comprehensiveness of the search

The comprehensiveness of the search strategy will be assessed by comparing results to a list of benchmark articles. This test-list of articles is the same used for the scoping exercise (see Additional file 1: Scoping). The ten papers in the list were deemed by the advisory group as paragon examples of articles that are relevant to the primary and secondary questions. The list covers a range of authors and topics including articles on environmental impacts, management, governance, consumer awareness and the developing world, including quantitative analyses of stock assessments to more qualitative understandings of stakeholder perceptions of the program.

Publication data bases

The following online databases will be searched:

• 1. Clarivate Analytics Web of Science™ Core Collection http://apps.webofknowledge.com/.

• 2. Elsevier’s SCOPUS http://www.elsevier.com/online-tools/scopus.

• 3. AGRIS database http://agris.fao.org/agris-search/index.do.

Supplementary searches

In addition to searches in databases, hand searching in MSC’s own grey literature and other organisational websites will be done (including WWF, FAO and ISEAL Alliance). Literature will also be provided directly by members of the advisory group and call for extra literature through their own contacts and other related stakeholders. The references from other relevant reviews [17,18,19] will be also be included if they have not already been found though the above searching.

Search engines

The same search terms will also be applied in Microsoft Academic (http://academic.research.microsoft.com) and Google Scholar (https://scholar.google.co.uk) in order to source additional grey literature not found elsewhere [24]. For these searches, ‘Harzing’s publish or perish’ plug-in will be used to extract the first 250 entries, following the recommendation by Haddaway et al. [24]. Publish or Perish is a software program that retrieves and analyses academic citations. It uses a variety of data sources (incl. Google Scholar) to obtain raw citations. Harzing’s Publish or Perish is available at: https://harzing.com/resources/publish-or-perish. Articles found with Publish or Perish will be screened in a similar fashion to the databases. Changes in the above search string will be formatted to be compatible with the search engines.

Article screening and study eligibility criteria

Screening process

Following the completion of all searches, these will be imported into Colandr (a product of a collaborative partnership between the Science for Nature and People Partnership Evidence-Based Conservation working group, DataKind, and Conservation International—https://www.colandrapp.com/) to allow screening by multiple screeners. Colandr provides a platform for all systematic mapping requirements, from title and abstract screening to meta-data extraction, and applies machine learning and natural language processing algorithms to sort articles according to relevance which can reduce the time taken to screen compared with traditional methods [25].

An exercise to measure inter-rater agreement using Cohen’s kappa [26] will be undertaken before title, abstract and full-text screening and meta-data extraction [27]. A random sample of 10% of the literature will be screened by the reviewers according to the inclusion and exclusion criteria detailed below and their results directly compared. Where conflicts exist, they will be discussed and rectified to ensure all reviewers agree. Once a Kappa score of ≥ 0.6 has been achieved, individual screening will commence.

In order to ensure consistency of decisions, minimum of 10% of articles will be reviewed by two coders at the title, abstract and full text screening levels. Where disagreements occur, the coders will discuss them and come to a consensus to either include or exclude articles. As this iteratively occurs through the different screening levels, the coders’ decision-making and reasoning processes are likely to converge to increase the level of consistency. All screening (at title, abstract and full-text levels) will be performed by 2 non-MSC coders.

It is recognised that the MSC, on occasion, plays a significant role in individual fisheries management and policy debates. For example, MSC representatives may be present at annual stock assessment meetings and discussions may be framed around MSC relevant issues. This may be captured in the meeting minutes. It is acknowledged while these minutes and similar documents may be an important resource and provide a holistic understanding of the MSC and its impacts, they cannot be included in this systematic map due to concerns regarding sampling bias, veracity, transparency and repeatability.

All included will be recorded and included in the final report. A list of articles excluded at full text with reasons will also be available.

Demonstrating procedural independence

Readers that have authored articles that are identified in the search process will not be able to read these or comment on their inclusion or exclusion. When this occurs, the author will refer the article to another reviewer.

Eligibility criteria

Eligible populations or subjects

Relevant subjects are fisheries, companies, governments, communities and biological populations or ecosystems that have been or are potentially being certified or involved in the certification process. Additionally, those fisheries, companies, governments, communities and biological populations or ecosystems potentially affected through certification of other subjects are also eligible populations.

Eligible intervention

The eligible intervention in question is the MSC programme. This can include a certified fishery or chain of custody actors and any subjects potentially becoming certified according to the fishery or chain of custody standard (although these will be noted separately).

Eligible comparators

The absence of MSC certification (or potential MSC certification) either between fisheries, governments, sites or groups, and/or over time will be included. Articles that do not have a comparator will be included but noted separately. This will be done to include social science and political science studies and others that may lack a comparator, but whose outcomes are still of importance to this study.

Eligible outcomes

Any outcome related impacts on ecosystems or biodiversity, trade, socio-economic outcomes, governance, management or policy will be included. Examples may include: the effects of MSC certification on stock health, fishing mortality or gear use; the effects of the MSC certification process on interactions between stakeholders (i.e. changes in partnerships or conflict) or changes in consumer awareness regarding environmental sustainability in the marine realm or changes in consumers’ willingness to pay for MSC eco-labelled products.

Eligible types of study design

All types of study design will be included with the study type recorded. Only literature that does not pass the above PICO specifications, language and time limitations will be excluded. This will be done to get an understanding of the evidence generated about the MSC.

Study validity assessment

No formal, systematic study validity assessment of the included studies will be undertaken. This is not mandatory for systematic maps [28]. However, metadata pertaining to study design and data collection methods will be extracted.

Data coding strategy

Following full text screening, meta-data will be extracted from the included studies. All data extraction will be performed in Colandr using a data-extraction form developed in the online platform according to the following main categories and labels:

• 1. Bibliographic information (title, authors, date, journal, year).

• 2. Literature type (grey literature, journal article, conference proceedings etc).

• 3. Discipline (political science, economics, fisheries science etc).

• 4. Geographic information (country, continent, coordinates, development level based on the world bank list of economies).

• 5. Type of study design, data collection methods and quantitative/qualitative.

• 6. Fishery information (if particular fishery/fisheries are identified. E.g. certified fishery name, species, gear, sea, FAO major fishing area, fisheries scale).

• 7. Study focus (e.g. fish stocks, habitats and ecosystems, transparency in the supply chain etc).

• 8. Intervention type (certified fishery, certified CoC actor, prospective certified entities, fishery or CoC actors in assessment).

• 9. Outcome (as described in the study).

• 10. MSC impact (presence of MSC impact, description of specific MSC impact, whether the impact is intended/unintended according to the MSC’s standards and whether the evidence for the impact is direct or indirect (proxy/circumstantial).

• 11. MSC impact tested against the MSC’s intent (whether the study examines MSC’s impact or credibility against MSC’s own Theory of Change and/or vision and mission, and/or standards. I.e. the study does not test the MSC’s impacts against aspects outside of the program’s remits).

For a detailed overview of the labels, descriptions, input methods and rationales for the data-extraction form (including screen shots of the Colandr form) see Additional file 2: Data extraction sheet. This extraction form has been piloted using the initial scoping exercise test searches performed using web of science (01/06/2018) that recovered 162 articles. For the pilot, 10 articles were coded by 2 non-MSC and 2 MSC staff using a pilot data extraction sheet (see Additional file 3: Pilot data extraction sheet). The pilot testing allowed the authors to refine, streamline and reduce ambiguity in the final data-extraction sheet.

Before full data coding commences, an exercise to measure inter-rater agreement using Cohen’s kappa [26] using a random sample of 10% of the included articles will be performed. During this exercise any inconsistencies will be discussed between all coders and agreements made by consensus. The exercise will be repeated until a Kappa score of ≥ 0.6 has been achieved, after which individual coding will take place. This will not only ensure inter-rater agreement but also the consistency and repeatability of the study.

During individual coding where data is missing, unclear or ambiguous, the review team will search for additional information and make changes according to consensus. Where data is taken from elsewhere references to the extra sources will be cited to ensure repeatability. Additionally, any uncertainties and issues that arise during the data extraction process will be flagged by the coder. Flagged issues will be dealt with in semi-regular meetings and decisions to rectify the problem made unanimously. This will further ensure consistency in coding. To help with repeatability these decisions will be documented and made available in an additional file in the final systematic map report. Coders that have authored included papers will not code these articles. Data extraction will be performed by 2 non-MSC coders and 2 MSC staff. To reduce the risk of partiality the majority (up to 75%) of meta-data extraction will be performed by non-MSC staff.

The final extracted meta-data will be made included with the final report as additional files.

Study mapping and presentation

The final map and narrative will be described in accordance with the primary and secondary questions (see above). Further discussion will explain the relevance and highlight the knowledge gaps, clusters and areas for future research and their relation to the MSC’s monitoring and evaluation programme. To identify knowledge gaps and clusters cross tabulations of key variables will be used to create heatmaps and highlight areas with high and low concentrations of studies. The advisory group plans to meet once the results have been collected to set arbitrary but specific cut off points which could then be used to identify study areas or topics that lack evidence and are poorly studied areas. The same will be done to identify knowledge clusters; topics or study areas deemed to have sufficient studies to allow for meaningful synthesis. Figures describing the outcomes for each of the secondary questions will also be included (e.g. a bar graph showing the number of studies per country or number of studies per certified species). This will all be presented in a systematic map that will include this as well as the full methodology undertaken and links (via the additional supporting files) to a full database of all included studies with their meta-data.