History and Background 

Marine vessel traffic is having a negative anthropogenic impact on marine environments and marine mammals, such as killer whales and other cetaceans, which experience growing pressures by the increasing and constantly changing diverse anthropogenic ocean use and activities performed in coastal areas.

The underwater sound sources caused by boats, naval mid-frequency sonars and air-guns for marine geophysical prospecting, responsible for the noise pollution generated in underwater environments where sound travels five times faster than in air [1], have been associated with marine species behavioral disturbance and marine mammal vocalization interference.

According to research performed by the University of Victoria in Canada [6], marine vessel traffic has seen an increase over the last 30 years in the Canadian Arctic and is expected to increase with faster rhythms as sea-ice extent is declining due to climate change. The continued increase of ice-free waters has led to a significant impact on shipping activities in the Canadian Arctic with the number of vessels which are heading to or from Arctic ports to be increasing dramatically, including ships responsible for oil and gas exploration activities, cruise ships, military vessels, research boats and even sight-seeing whale watching touristic smaller vessels.

The side effects for the Arctic, a place that until recently has functioned as an acoustic refuge from industrial noise for the marine life, vary from:

• Limiting whales’ and other cetaceans’ listening and acoustic space; sound becomes vital to the survival of marine mammals in that they use it to detect and navigate their environment, locate predators and prey and communicate with each other.
• The increase in low frequency chronic noise emitted from vessels may have wide ranging masking effects on cetaceans.
• Marine mammals and especially whales are also at risk of ship strikes that may result in significant injury or even mortality. 

Therefore, a number of efforts have been introduced towards investigation and exploration of possible planned management scenarios, with the goal of mitigating those negative impacts on locally important marine species. An efficient study of marine traffic can help inform public policies being in effect and conservation efforts.

In this blog post we are going to look into the current suite of research proposals and tools being implemented to automatically detect marine vessels of various sizes under diverse conditions by the University of Victoria Canada (UVIc) in collaboration with the Canadian Wildlife Service (CWS), and Heriot Watt University of Scotland using:

• Data analysis techniques for hydro acoustic data and
• Artificial Intelligence (AI) techniques such as Machine Learning and Deep Learning algorithms for time series analysis.

What can be done?

One of the things that marine environmental protection authorities regularly require is the implementation of mitigation measures regarding vessel speed reduction, shutdown of acoustic sources in sensitive zones or predefined marine protection areas where marine mammals such as killer whales are sighted and creation of restriction zones. To make sure the mitigation measures will be successful, what is first needed is to reliably identify the areas that are frequently used by marine mammals (including important areas for feeding or calving) who are “the recipients”, and those areas commonly frequented by marine vessels, “the stressors”.

Ship-based marine mammal observers (MMOs) are the most commonly implemented detection methods for monitoring sensitive marine protected zones. However, the detection of marine mammals and especially whales in the open ocean is challenging with human observer’s performance to be variable based on environmental conditions such as daylight, fog, sea waves and wind force, as well as the physical state of the observer and the animal behavior [2]. MMOs being a person-intensive activity is usually used in combination to hydrophone arrays that can run day and night, passive acoustic monitoring (PAM), turning MMOs in a more effective combination that can be used during night time as well [3], with PAM being also used under-water to avoid issues related to surface detection. However, the downside of PAM is that they rely on animals vocalizing and little background noise that will mask the ability of the hydrophone to detect the animals.

On the other hand, a common tool used for detection of marine vessels is Automatic Identification System (AIS) data that can be used to track and analyze stressors such as noise and vessel-strike. Yet, as AIS has not been designed for the purpose of mitigation and conservation but mainly for marine vessels detection to ensure navigational safety, it can often ignore the impact of smaller vessels that usually do not broadcast AIS data. 

Such a scenario of the impact and detection of small marine vessels has been explored by the UVIc in collaboration with the CWS, and Heriot Watt University in the densely populated coastal regions of the Salish Sea in the Pacific that happens to host a critical habitat for endangered Southern Resident Killer Whales (SRKW), with the area being highly trafficked by small whale-watching, fishing and research boats. These vessels emit high acoustic energy in the mid and high-frequency ranges, forcing negative anthropogenic impact for the sensitive marine species [4]. In their research effort towards automatic identification of small vessel boats they have followed an approach of analyzing monitoring images originating from on land based cameras in their goal to enable marine conservation, research and observation of the impacts of smaller vessels on killer whales. Among the challenges they have described in this effort was the manual review of thousands of images as part of the large datasets captured by the land-based cameras that have been installed in the coastal areas being configured with a high frequency of time lapse captures.

The research team came to the proposal of a prototype hybrid system solution for the detection of  marine vessels that combines state-of-the-art pre-trained deep learning (DL) object detectors and a novel detector of small marine vessels (DSMV) [4]. 

Let’s understand more on that!

Using Deep Learning (DL)-based object detectors and DSMV

Object detectors are a family of techniques based on computer vision and image processing that deals with the detection of instances of objects falling under a particular classification within digital images or video time lapses. Most recent work on deep learning (DL)-based object detectors identify well medium and large-sized vessels [4], yet the smaller vessels often get missed by pre-trained end-to-end object detectors. For example, medium and large-sized boats are efficiently identified by pre-trained end-to-end object detectors such as RetinaNet, Faster R-CNN and Cascade R-CNN [4].

The DSM uses a short time series of images and a novel bi-directional Gaussian Mixture Model technique to determine motion in combination with context-based filtering and a DL-based image classifier. The term small refers to vessels that have approximately 80 pixels of area, while medium and large vessels are those that occupy approximately 800 and 1, 500 pixels of area, respectively.

 The image below summarizes the proposed hybrid detection approach.

DSMV considers a temporal window composed by three 5-second-apart images of the same location (referred to as “BWD”, “MID”, and “FWD” images). The four assumptions that help distinguish small marine vessels from false positives have been incorporated as contextual information to the DSMV [4]: 

(i) vessels that appear small on monitoring images are those farther away from the camera, and thus they should only move horizontally in a 15-second time window; 

(ii) a boat identified in the MID image will remain inside a sub-region of this image bounded by the same boat identified in the BWD and FWD images; 

(iii) the boat in the MID image is going to be positioned roughly in the middle of the aforementioned sub-region;

(iv) the three sightings associated with the same boat in the three images should present a similar visual appearance.

In the proposed system an exclusive GMM is created for each group of three images (i.e. BWD, MID and FWD), allowing for systems with low monitoring frequency to still use the proposed DSMV. During the forward motion, the BWD and MID images are used to model the background, thus detecting the foreground on the FWD sample. After a morphological filtering step, the sets of connected components from each motion that represent vessel candidates are used to delimit bounding boxes (BB) in the FWD and BWD images named forward-motion bounding boxes (FMBB) and backward-motion bounding boxes (BMBB). Then the coordinates of the FMBB are used to determine a horizontal detection band where a template matching is performed. The vessel sighting in the MID image (MIBB) is expected to be placed roughly between the FMBB and the BMBB.

In order to further refine the list of vessel candidates, a custom DL-based image classifier is used at the end of the DSMV to classify each individual bounding box in a group of BMBB-MIBB-FMBB. A group of BMBB-MIBB-FMBB is only deemed as valid if the content of all three BBs is classified as an object of the vessel class. The custom-trained DL-based system performs a binary classification where each image patch is classified as either background or vessel. These manually-curated image patches are resized to comply with each classifier’s CNN input requirements and used in the training routine.

Given the efficacy of object detectors for large and medium sized vessels the team has combined the output of object detectors and the DSMV (small-sized boats) into a robust hybrid detector system that is now able to identify boats of variable sizes.

Smaller vessels are challenging to end-to-end object detectors because their feature extractors are based on multiple layers of sequential convolutions that generate feature maps of progressively smaller spatial dimensions [4]. Small visual targets in the original input tend to disappear during the feature extraction process (i.e., they are represented by less than a pixel in the feature maps at a given depth in the CNN), which prevents them from being localized and classified. The figure below illustrates the results of the method proposed in [3], which combines the capabilities of pre-trained end-to-end object detectors and the novel DSMV to automatically identify vessels of diverse scales.

“The automatic identification of marine vessels in environmental monitoring images includes a number of challenges, including the vessel’s different sizes , shapes, colors, as well as  the scene visibility and ever-changing appearance of the water surface.“

Small Size (vessel) 

Up: a single small-vessel represents an extremely challenging identification task. Down: in this image where three input frames were merged, one can use the perceived motion as a cue for the presence of a small marine vessel.

                                                      Shape Variability                                                                   

Surface of the water (calm vs agitated)

Weather Visibility

Final system achieved

The proposed system was implemented using Python and PyTorch and object detector implementations from Detectron2 developed by Facebook research. Extensive experimental results obtained on their captured datasets of images containing boats of various sizes show that the proposed approach comfortably outperforms other popular state-of-the-art object detectors [4]. The proposed detector solution is able to serve the needs of real-world automated processing of governance in critical habitats such as the Salish Sea and its timely interpretation of monitoring data allows the support of marine conservation and research efforts. 

Future Actions towards further Mitigation

The University of Victoria is currently partnering with the Wildlife Conservation Society Canada (WCS) and Environment and Climate Change Canada (ECCC) to design a novel system for the automatic  detection of marine vessels and mammals in the western Canadian Arctic using a hybrid of visual and passive acoustics data. They foresee that the contributions of this project will provide insights that will enable the assessment of the impact of increased vessel traffic on local marine species such as beluga whales. This is in an effort to better quantify, monitor and study their ecological impact that can create awareness to the public and proceed with informed conservation policies and planning mitigation.

We can clearly see in all those efforts that the impact of Artificial Intelligence (AI) such as the use of deep learning classification techniques for analyzing visual or acoustic data is going to help on delivering more reliable, time- and cost-efficient solutions to traditional MMO activities.

AI to the rescue again !!

For deeper learning you can look into the following resources …

[1] Abdulla, Ameer, ed. Maritime traffic effects on biodiversity in the Mediterranean Sea. Volume 1: review of impacts, priority areas and mitigation measures. Vol. 1. IUCN, 2008

[2] Zitterbart, Daniel & Smith, Heather & Flau, Michael & Richter, Sebastian & Burkhardt, Elke & Beland, Joe & Bennett, Louise & Cammareri, Alejandro & Davis, Andrew & Holst, Meike & Lanfredi, Caterina & Michel, Hanna & Noad, Michael & Owen, Kylie & Pacini, Aude & Boebel, Olaf. (2020). Scaling the Laws of Thermal Imaging–Based Whale Detection. Journal of Atmospheric and Oceanic Technology. 37. 10.1175/JTECH-D-19-0054.1

[3] Heather R. Smith, Daniel P. Zitterbart, Thomas F. Norris, Michael Flau, Elizabeth L. Ferguson, Colin G. Jones, Olaf Boebel, Valerie D. Moulton, A field comparison of marine mammal detections via visual, acoustic, and infrared (IR) imaging methods offshore Atlantic Canada, Marine Pollution Bulletin, Volume 154, 2020, 111026,ISSN 0025-326X, https://doi.org/10.1016/j.marpolbul.2020.111026

[4] Porto Marques, Tunai & Albu, Alexandra & O’Hara, Patrick & Serra Sogas, Norma & Morrow, Ben & Mcwhinnie, Lauren & Canessa, Rosaline. (2020). Size-invariant Detection of Marine Vessels From Visual Time Series. 

[5] Zitterbart DP, Kindermann L, Burkhardt E, Boebel O (2013) Automatic Round-the-Clock Detection of Whales for Mitigation from Underwater Noise Impacts. PLoS ONE 8(8): e71217. ”https://doi.org/10.1371/journal.pone.0071217

[6] McWhinnie, Lauren H., et al. “Vessel traffic in the Canadian Arctic: Management solutions for minimizing impacts on whales in a changing northern region.” Ocean & Coastal Management 160 (2018): 1-17.

[7] Huntington, H.P., Daniel, R., Hartsig, A., Harun, K., Heiman, M., Meehan, R., Noongwook, G., Pearson, L., Prior-Parks, M., Robards, M., Stetson, G., 2015. Vessels, risks and rules: planning for safe shipping in Bering Strait. Mar. Policy 51, 119–127. http://dx.doi.org/10.1016/j.marpol.2014.07.027

[8] Daniel P. Zitterbart, Lars Kindermann, Elke Burkhardt, Olaf Boebel, Automatic Round-the-Clock Detection of Whales for Mitigation from Underwater Noise Impacts

[9] Size-invariant Detection of Marine Vessels From Visual Time Series, Porto Marques, Tunai, (Presentation slides,  url-link)

[9] Photography @Craig Parry Photography, url-link

[10] Photography @chasedekkerphotography url-link

[11] Photography @GeorgeCarbus url-link

        Special thanks to contributors:

• Tunai P. Marques, PhD Candidate University of Victoria, BC, Canada
• Dr. Lauren McWhinnie, Assistant Professor at Institute of Life and Earth Sciences, School of Energy, Geoscience, Infrastructure and Society, Heriot-Watt University & Adjunct Assistant Professor, Department of Geography, University of Victoria
• Patrick D. O’Hara, PhD., Seabird Conservation, ECCC-CWS, Institute of Ocean Sciences & Adjunct Associate Professor, Department of Geography, University of Victoria