Digital Stack Photography and Its Applications

Abstract

This work centers on digital stack photography and its applications.A stack of images refer, in a broader sense, to an ensemble ofassociated images taken with variation in one or more than one various values in one or more parameters in system configuration or setting.An image stack captures and contains potentially more information thanany of the constituent images. Digital stack photography (DST)techniques explore the rich information to render a synthesized imagethat oversteps the limitation in a digital camera's capabilities.This work considers in particular two basic DST problems, which hadbeen challenging, and their applications. One is high-dynamic-range(HDR) imaging of non-stationary dynamic scenes, in which the stackedimages vary in exposure conditions. The otheris large scale panorama composition from multiple images. In thiscase, the image components are related to each other by the spatialrelation among the subdomains of the same scene they covered andcaptured jointly. We consider the non-conventional, practical andchallenge situations where the spatial overlap among the sub-images issparse (S), irregular in geometry and imprecise from the designedgeometry (I), and the captured data over the overlap zones are noisy(N) or lack of features. We refer to these conditions simply as theS.I.N. conditions.There are common challenging issues with both problems. For example,both faced the dominant problem with image alignment forseamless and artifact-free image composition. Our solutions to thecommon problems are manifested differently in each of the particularproblems, as a result of adaption to the specific properties in eachtype of image ensembles. For the exposure stack, existingalignment approaches struggled to overcome three main challenges:inconsistency in brightness, large displacement in dynamic scene andpixel saturation. We exploit solutions in the following threeaspects. In the first, we introduce a model that addresses and admitschanges in both geometric configurations and optical conditions, whilefollowing the traditional optical flow description. Previous modelstreated these two types of changes one or the other, namely, withmutual exclusions. Next, we extend the pixel-based optical flow modelto a patch-based model. There are two-fold advantages. A patch hastexture and local content that individual pixels fail to present. Italso renders opportunities for faster processing, such as viatwo-scale or multiple-scale processing. The extended model is thensolved efficiently with an EM-like algorithm, which is reliable in thepresence of large displacement. Thirdly, we present a generativemodel for reducing or eliminating typical artifacts as a side effectof an inadequate alignment for clipped pixels. A patch-based texturesynthesis is combined with the patch-based alignment to achieve anartifact free result.For large-scale panorama composition under the S.I.N. conditions, wehave developed an effective solution scheme that significantly reducesboth processing time and artifacts. Previously existing approaches canbe roughly categorized as either geometry-based composition or featurebased composition. In the former approach, one relies on preciseknowledge of the system geometry, by design and/or calibration. Itworks well with a far-away scene, in which case there is only limitedvariation in projective geometry among the sub-images. However, thesystem geometry is not invariant to physical conditions such asthermal variation, stress variation and etc.. The composition withthis approach is typically done in the spatial space. The otherapproach is more robust to geometric and optical conditions. It workssurprisingly well with feature-rich and stationary scenes, not wellwith the absence of recognizable features. The composition based onfeature matching is typically done in the spatial gradient domain. Inshort, both approaches are challenged by the S.I.N. conditions. Withcertain snapshot data sets obtained and contributed by Brady et al, these methods either fail in composition or render images withvisually disturbing artifacts. To overcome the S.I.N. conditions, wehave reconciled these two approaches and made successful andcomplementary use of both priori and approximate information aboutgeometric system configuration and the feature information from theimage data. We also designed and developed a software architecturewith careful extraction of primitive function modules that can beefficiently implemented and executed in parallel. In addition to amuch faster processing speed, the resulting images are clear andsharper at the overlapping zones, without typical ghosting artifacts.