Drones and Machine Learning for Marine Animal Behavior Analysis

Abstract

Understanding the behavior of marine animals is critical to effective management especially as they fall under increasing anthropogenic pressures. Recent advances in two technologies, drones and machine learning offer versatile, data driven, automatable solutions capable of effective collection and analysis of large datasets. In this paper I illustrate how pose estimation as an effective machine learning based solution for analyzing marine animal behavior. This study investigates pose estimations use on drone imagery due to its rising prevalence in marine science and prior combination with pose estimation in our lab. As initial work at our lab has investigated the use of pose estimation on marine mammal datasets and my goal is to expand on these efforts and build an overview of both technologies integration for researchers interested working with them. In the present study I use a collection of shark video taken by myself and other Duke researchers locally off the Rachel Carson Reserve on the North Carolina coast as demonstration and to help build a catalog of models and best practices for use of pose estimation on different taxa. This paper will provide an overview of drones and pose estimation including Social LEAP Estimates Animal Poses (SLEAP), a pose estimation framework which has proven to have good potential in marine science. SLEAP was chosen due to its accessibility, versatility and tracking algorithm which allows multiple subjects to be tracked and analyzed at the same time. The latter is a major steppingstone for pose estimation software as past projects may have been able to identify multiple individuals in one frame but not be able to keep track of who is who across thousands of frames of video. Covered topics will include: 1. Technical overview of drones and pose estimation. 2. Data collection 3. Using pose estimation a. Model types and programming 4. Data export and processing 5. Analysis 6. Conclusions on using pose estimation in marine science and future work. After data export, a novel solution will also be assessed for compensating for camera movement, in this case a moving drone, which has proven to be one of the biggest roadblocks of using SLEAP, which was developed for processing stationary video. This solution processes data in a way that is plug and play with existing analytical methods and will be open source.