Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second.

Abstract

Wide field of view microscopy that can resolve 3D information at high speed and spatial resolution is highly desirable for studying the behaviour of freely moving model organisms. However, it is challenging to design an optical instrument that optimises all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising on speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over an area of 135 cm2, achieving up to 230 frames per second at spatiotemporal throughputs exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The resulting reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with collections of several freely behaving organisms, including ants, fruit flies, and zebrafish larvae.

Department

Description

Provenance

Citation

Published Version (Please cite this version)

10.1038/s41566-023-01171-7

Publication Info

Zhou, Kevin C, Mark Harfouche, Colin L Cooke, Jaehee Park, Pavan C Konda, Lucas Kreiss, Kanghyun Kim, Joakim Jönsson, et al. (2023). Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second. Nature photonics, 17(5). pp. 442–450. 10.1038/s41566-023-01171-7 Retrieved from https://hdl.handle.net/10161/30703.

This is constructed from limited available data and may be imprecise. To cite this article, please review & use the official citation provided by the journal.

Scholars@Duke

Zhou

Kevin Zhou

Affiliate
Bagnat

Michel Bagnat

Nanaline Duke Distinguished Professor
Horstmeyer

Roarke Horstmeyer

Assistant Professor of Biomedical Engineering

Roarke Horstmeyer is an assistant professor within Duke's Biomedical Engineering Department. He develops microscopes, cameras and computer algorithms for a wide range of applications, from forming 3D reconstructions of organisms to detecting neural activity deep within tissue. His areas of interest include optics, signal processing, optimization and neuroscience. Most recently, Dr. Horstmeyer was a guest professor at the University of Erlangen in Germany and an Einstein postdoctoral fellow at Charitè Medical School in Berlin. Prior to his time in Germany, Dr. Horstmeyer earned a PhD from Caltech’s electrical engineering department in 2016, a master of science degree from the MIT Media Lab in 2011, and a bachelors degree in physics and Japanese from Duke University in 2006.


Unless otherwise indicated, scholarly articles published by Duke faculty members are made available here with a CC-BY-NC (Creative Commons Attribution Non-Commercial) license, as enabled by the Duke Open Access Policy. If you wish to use the materials in ways not already permitted under CC-BY-NC, please consult the copyright owner. Other materials are made available here through the author’s grant of a non-exclusive license to make their work openly accessible.