Sampling Strategies and Neural Processing for Array Cameras

Artificial intelligence (AI) reshapes computational imaging systems. Deep neural networks (DNN) not only show superior reconstruction performance over conventional ones handling the same sampling systems, these new reconstruction algorithms also call for new sampling strategies. In this dissertation, we study how DNN reconstruction algorithms and sampling strategy can be jointly designed to boost the system performance.

First, two DNNs for sensor fusion tasks based on convolutional neural networks (CNN) and transformers are proposed. They are able to fuse frames with different resolution, different wave band, or different temporal window. The amount of frames can also vary, showing great flexibility and scalability. A reasonable computational load is achieved by a proper receptive field design balancing the flexibility and complexity. Visual pleasing reconstruction results are achieved.

Then we demonstrate how DNN reconstruction algorithms favor certain sampling strategy over another, with snapshot compressive imaging (SCI) task as an example. Using synthetic datasets, we compare quasi-random coded sampling and multi-aperture multi-scale manifold sampling under DNN reconstruction. The latter sampling strategy requires much simpler physical setup, yet gives comparable, if not better, reconstruction image quality.

At the end, we design and build a multifocal array camera fitting the DNN reconstruction. With commercial on-the-shelf cameras and lenses, the array camera achieves a nearly 70 degree field of view (FoV), a 0.1m - 17.1m depth of field (DoF), and the ability to resolve objects with 2mm granularity. One final output image contains about 33M RGB pixels.

Overall, we explore the joint design of DNN reconstruction algorithms and physics sampling. With our research, we hope to develop more compact, more accurate, and larger covering range computational imaging systems.