Sampling in Computational Cameras

Abstract

<p>This dissertation contributes to computational imaging by studying the intersection of sampling and artificial intelligence (AI). It has been demonstrated that AI shows superior performance in various image processing problems, ranging from super- resolution to classification. In this work we demonstrate that combining AI with intelligent data sampling enables new camera capabilities.We start with traditional image signal processing (ISP) in digital cameras and show that AI has significantly improved the performance of ISP functions, such as demosaicing, denoising, and white balance. We than demonstrate a deep-learning (DL)-based image signal processor that regroups the ISP functions and achieves fast image processing with an end-to-end network. We further study the image compression strategies and show that AI is also a helpful tool for imaging system design. Following the study on image processing, we turn to the camera autofocus con- trol. With the demonstration of a DL-based autofocus pipeline and saliency de- tection network, we show that AI achieves 5 - 10x faster autofocus compared to traditional contrast maximization and allows content-based autofocus control. We also demonstrate an all-in-focus imaging pipeline to produce all-in-focus images or videos. This shows that AI extends the concept of camera control from optimizing an instantaneous image to producing the control trajectory that optimizes the sampling eciency and long-term image or video quality. Next we consider coherent phase retrieval. We first study the Fisher information and the Cram ́er-Rao lower bound on the mean squared error of coherent signal estimation from the squared modulus of its linear transform. Then we demonstrate two coding strategies to achieve optimal phase retrieval, and we use simulations to show practical implementations of these strategies. These simulation take the advantage of well-developed deep learning libraries. Finally we focus on Fourier ptychography, a technique combining aperture synthe- sis and phase retrieval. We build a snapshot ptychography system using a camera array and deep neural estimation, which achieves 6.7ˆ improvement in resolution compared to a single camera. We also present simulations considering various aper- ture distributions and multiple snapshots to show design considerations of such as system.</p>