Digital Stack Photography and Its Applications

This work centers on digital stack photography and its applications.

A stack of images refer, in a broader sense, to an ensemble of

associated images taken with variation in one or more than one various

values in one or more parameters in system configuration or setting.

An image stack captures and contains potentially more information than

any of the constituent images. Digital stack photography (DST)

techniques explore the rich information to render a synthesized image

that oversteps the limitation in a digital camera's capabilities.

This work considers in particular two basic DST problems, which had

been challenging, and their applications. One is high-dynamic-range

(HDR) imaging of non-stationary dynamic scenes, in which the stacked

images vary in exposure conditions. The other

is large scale panorama composition from multiple images. In this

case, the image components are related to each other by the spatial

relation among the subdomains of the same scene they covered and

captured jointly. We consider the non-conventional, practical and

challenge situations where the spatial overlap among the sub-images is

sparse (S), irregular in geometry and imprecise from the designed

geometry (I), and the captured data over the overlap zones are noisy

(N) or lack of features. We refer to these conditions simply as the

S.I.N. conditions.

There are common challenging issues with both problems. For example,

both faced the dominant problem with image alignment for

seamless and artifact-free image composition. Our solutions to the

common problems are manifested differently in each of the particular

problems, as a result of adaption to the specific properties in each

type of image ensembles. For the exposure stack, existing

alignment approaches struggled to overcome three main challenges:

inconsistency in brightness, large displacement in dynamic scene and

pixel saturation. We exploit solutions in the following three

aspects. In the first, we introduce a model that addresses and admits

changes in both geometric configurations and optical conditions, while

following the traditional optical flow description. Previous models

treated these two types of changes one or the other, namely, with

mutual exclusions. Next, we extend the pixel-based optical flow model

to a patch-based model. There are two-fold advantages. A patch has

texture and local content that individual pixels fail to present. It

also renders opportunities for faster processing, such as via

two-scale or multiple-scale processing. The extended model is then

solved efficiently with an EM-like algorithm, which is reliable in the

presence of large displacement. Thirdly, we present a generative

model for reducing or eliminating typical artifacts as a side effect

of an inadequate alignment for clipped pixels. A patch-based texture

synthesis is combined with the patch-based alignment to achieve an

artifact free result.

For large-scale panorama composition under the S.I.N. conditions, we

have developed an effective solution scheme that significantly reduces

both processing time and artifacts. Previously existing approaches can

be roughly categorized as either geometry-based composition or feature

based composition. In the former approach, one relies on precise

knowledge of the system geometry, by design and/or calibration. It

works well with a far-away scene, in which case there is only limited

variation in projective geometry among the sub-images. However, the

system geometry is not invariant to physical conditions such as

thermal variation, stress variation and etc.. The composition with

this approach is typically done in the spatial space. The other

approach is more robust to geometric and optical conditions. It works

surprisingly well with feature-rich and stationary scenes, not well

with the absence of recognizable features. The composition based on

feature matching is typically done in the spatial gradient domain. In

short, both approaches are challenged by the S.I.N. conditions. With

certain snapshot data sets obtained and contributed by Brady et al,

these methods either fail in composition or render images with

visually disturbing artifacts. To overcome the S.I.N. conditions, we

have reconciled these two approaches and made successful and

complementary use of both priori and approximate information about

geometric system configuration and the feature information from the

image data. We also designed and developed a software architecture

with careful extraction of primitive function modules that can be

efficiently implemented and executed in parallel. In addition to a

much faster processing speed, the resulting images are clear and

sharper at the overlapping zones, without typical ghosting artifacts.