Ocean observing data is growing at an accelerating pace, however, more is needed to enable well informed, ocean conservation and management. Many conventional workflows for marine data processing in ocean conservation and research are not automated, inhibiting the productivity of researchers and managers. This year SOI focused on projects leveraging the utilization of Falkor and ROV SuBastian that worked towards automating marine data analysis workflows to accelerate the understanding of our rapidly changing marine habitats and multiply the conservation benefits. 

SOI-SUPPORTED SCIENTIFIC IMAGE ANNOTATION SOFTWARE BECOMES A NATIONAL STANDARD

Squidle+ is a web-based open source software for the annotation of marine scientific images, videos, and photomosaics. SOI has supported the development of this software since 2015 when it was initially deployed as part of a Falkor-hosted R&D project. The near-real-time crowd-sourced annotation of AUV acquired seafloor imagery has been further developed by SOI for shipboard ROV SuBastian annotation and event logging.

The enhanced version of Squidle+ has been adopted by the Australian National Environmental Science Program and Japan Agency for Marine Science and Technology as the preferred platform for annotating images. Other government agencies such as Canada’s Department of Fisheries and Oceans and U.S.’s National Oceanic and Atmospheric Administration are exploring using the platform for their image annotation purposes as well.

Ultimately, our goal is to use what we have learned to inform the design of fully autonomous robots that will one day explore the oceans of Europa and Enceladus.

          — Richard Camili, Principal Investigator

Navigator Officer Marissa Judkins works with the team performing final checks before launching TUNA-SAND. With its systems and navigation capabilities tested and confirmed, the AUV is ready to begin work at the sea floor.
ROV pilot Jason Rodriguez wears eye detection glasses, which record his eye motions and focus, as part of the data that will be used to train robotic algorithms.
Richard Camilli (Principal Investigator of the "New Approaches To Autonomous Exploration At The Costa Rican Shelf Break") and David Pinga assemble a Sloucm glider in R/V Falkor's wet lab.
COMPILING AND ANALYSING THE LARGEST KNOWN CENTIMETER-RESOLUTION MODEL OF SEAFLOOR

In August, Dr. Blair Thornton, University of Southampton, and his collaborators from the UK and Japan deployed two AUVs from Falkor to acquire 1.3 million seafloor images of changing microbial habitats surrounding methane seeps off the U.S. West Coast. The team used shipboard high performance computers to rapidly compose these images into the largest known centimeter-resolution color 3D model of 11.4 hectares of the seafloor. An unsupervised machine learning algorithm was used to cluster the images by visual similarity and locate continuously changing microbial hotspots on the seafloor. Detailed surveys were conducted using ROV SuBastian the following day.

This project demonstrated how throughput data processing and machine learning can multiply the productivity of marine scientists. It allowed the team to make quick and well-informed sampling decisions and conduct fine scale surveys to study habitats that would otherwise be virtually impossible to observe in detail. The resulting composite map was invaluable in planning operations, including the recovery of seafloor instruments, re-visiting active bubble plumes, and made the entire operation more efficient. Using AI, scientists have the ability to gather data and make decisions in expedition relevant time frames, prompting better use of the robotic systems deployed at sea.

A) 3D photogrammetric reconstruction of the Southern Hydrates Ridge (depth 780 m) that was generated by the AUV Ae2000f. The mapped area covers 11.8 ha at an average resolution of 6 mm and is the largest known expanse of seafloor that has been visually mapped in colour. The Falkor shown to scale in the top left of the panel. B) Outputs of an unsupervised classifier that was used to analyze the imagery between deployments and generated information summaries such as the image class matrix E). C) White bacterial mat that was identified by the classifier, which was subsequently surveyed by the AUV Tuna-sand at sub-millimeter resolution to gather more detailed information. The arrow in C) illustrates the direction from which the isometric view of the reconstruction is viewed in D).
TEACHING ROBOTS TO LEARN FROM DATA TO PLAN AND EXECUTE OCEAN SURVEYS

Falkor’s last expedition of 2018 hosted specialized teams, including computer scientists, roboticists, and oceanographic researchers from Woods Hole Oceanographic Institution (WHOI), MIT, NASA, ACFR, and the University of Michigan. The groups worked together, developing advanced AI for autonomous robotic exploration of remote uncharted environments. This technology will support future planetary exploration in space and in deep sea habitats, where due to communications latencies, robots need to make tactical decisions without human intervention.

One of the developed science software tools, the AI/machine learning-based autonomous planner “Spock,” was tested aboard Falkor in December. Deployed on the high performance computer, “Spock” parsed large volumes of diverse marine data from the ship’s and robots’ instruments, earlier surveys, and scientific annotations to suggest what new sites may be of potential interest for scientific surveys. Instead of traditional localization programming to avoid hazards, “Spock” let the shipboard researchers task the robots at much higher levels by simply asking them to “locate gas seeps”, “find rock formations”, or “survey biological hotspots.” All other mission details such as where to look for such features and how to survey them with available sensors were determined by AI.

The researchers led by Dr. Richard Camilli were fascinated to see the robotic systems demonstrating the ability to find and map rocky outcrops and biological hotspots without being directed where to search. Rapid, AI-driven data analysis more than doubled the utilization of robots during this expedition and allowed the researchers to survey 28,490 square kilometers of ocean within only nine days.

Dr. Richard Camilli directs the delicate deployment of a glider from Falkor's aft deck.