Mesoscopic photogrammetry with an unstabilized phone camera
Abstract
We present a feature-free photogrammetric technique that enables quantitative 3D mesoscopic (mm-scale height variation) imaging with tens-of-micron accuracy from sequences of images acquired by a smartphone at close range (several cm) under freehand motion without additional hardware. Our end-to-end, pixel-intensity-based approach jointly registers and stitches all the images by estimating a coaligned height map, which acts as a pixel-wise radial deformation field that orthorectifies each camera image to allow homographic registration. The height maps themselves are reparameterized as the output of an untrained encoder-decoder convolutional neural network (CNN) with the raw camera images as the input, which effectively removes many reconstruction artifacts. Our method also jointly estimates both the camera's dynamic 6D pose and its distortion using a nonparametric model, the latter of which is especially important in mesoscopic applications when using cameras not designed for imaging at short working distances, such as smartphone cameras. We also propose strategies for reducing computation time and memory, applicable to other multi-frame registration problems. Finally, we demonstrate our method using sequences of multi-megapixel images captured by an unstabilized smartphone on a variety of samples (e.g., painting brushstrokes, circuit board, seeds).
Type
Department
Description
Provenance
Citation
Permalink
Collections
Scholars@Duke
Kevin Zhou
Roarke Horstmeyer
Roarke Horstmeyer is an assistant professor within Duke's Biomedical Engineering Department. He develops microscopes, cameras and computer algorithms for a wide range of applications, from forming 3D reconstructions of organisms to detecting neural activity deep within tissue. His areas of interest include optics, signal processing, optimization and neuroscience. Most recently, Dr. Horstmeyer was a guest professor at the University of Erlangen in Germany and an Einstein postdoctoral fellow at Charitè Medical School in Berlin. Prior to his time in Germany, Dr. Horstmeyer earned a PhD from Caltech’s electrical engineering department in 2016, a master of science degree from the MIT Media Lab in 2011, and a bachelors degree in physics and Japanese from Duke University in 2006.
Sina Farsiu
I am the director of the Vision and Image Processing (VIP) Laboratory. Along with my colleagues, we investigate how to improve early diagnostic methods and find new imaging biomarkers of ocular and neurological diseases in adults (e.g. age-related macular degeneration, diabetic retinopathy, Glaucoma, Alzheimer) and children (e.g. retinopathy or prematurity). We also develop automatic artificial intelligence machine learning and deep learning algorithms to detect/segment/quantify anatomical/pathological structures seen on medical images.
On another front, we study efficient signal processing based methods to overcome the theoretical and practical limitations that constrain the achievable resolution of any imaging device. Our approach, which is based on adaptive extraction and robust fusion of relevant information from the expensive and sophisticated as well as simple and cheap sensors, has found wide applications in improving the quality of imaging systems such as ophthalmic SD-OCT, digital X-ray mammography, electronic and optical microscopes, and commercial digital camcorders. We are also interested in pursuing statistical signal processing based projects, including super-resolution, demosaicing, deblurring, denoising, motion estimation, compressive sensing/adaptive sampling, and sensor fusion.
Unless otherwise indicated, scholarly articles published by Duke faculty members are made available here with a CC-BY-NC (Creative Commons Attribution Non-Commercial) license, as enabled by the Duke Open Access Policy. If you wish to use the materials in ways not already permitted under CC-BY-NC, please consult the copyright owner. Other materials are made available here through the author’s grant of a non-exclusive license to make their work openly accessible.