Physical Designs in Artificial Neural Imaging

Artificial neural networks fundamentally shift the paradigm of computational imaging. Powerful neural processing is not only taking place of the conventional algorithms, but also embracing radical and physically plausible forward models that better sample the high dimensional light field. Physical designs of sampling in turn tailor simulation and neural algorithms for optimal inverse estimation. Sampling, simulation and neural algorithms as three essential components compose a novel imaging paradigm -- artificial neural imaging, in which they interact and improve themselves in an upward spiral.

Here we present three concrete examples of artificial neural imaging and the important roles physical designs play. In all-in-focus imaging, we see autofocus, sampling and fusion algorithms are redefined for optimizing the image quality of a camera with limited depth of field. Image-based neural autofocus acts 5-10x faster than traditional algorithms. The focus control based on the rule or reinforcement learning dynamically estimates the environment and optimizes the focus trajectory. Along with the neural fusion algorithm, the pipeline outperforms traditional focal stacking approaches in static and dynamic scenes. In scatter ptychography, we show imaging the secondary scatters reflected by a remote target under coherent illumination can create a synthetic aperture on the scatterer. The reconstruction of the object through phase retrieval algorithms can drastically exceed the resolution of directly viewing the target. In the lab experiment we demonstrate 32x resolution improvement relative to direct imaging using error-reduction and plug-and-play algorithms. In array camera imaging, we demonstrate heterogeneous multiaperture designs that have better sampling structures and physics-aware transformers for feature-based data fusion. The proposed transformer incorporates the physical information of the camera array as its receptive fields, demonstrating the superior ability of image compositing on array cameras with diverse resolutions, focal lengths, focal planes, color spaces, and exposures. We also demonstrate a scalable pipeline of data synthesis through computer graphics software that empowers the transformers.

The examples above justify artificial neural imaging and the physical designs interweaved. We expect better designs in sampling, simulation, neural algorithms and eventually better estimation of the light field.