Convex Optimization Methods for Structured Statistical Problems

Abstract

Convex optimization is an incredibly useful tool for a variety of mathematical settings and in this dissertation we elaborate on some of its uses for statistical problems. Many statistical problems require some type of structure on estimated parameters such as sparsity or restriction to a convex set. Convex optimization literature has extensively described how to handle constrained settings and regularization terms which induce structure in optima. In this dissertation we apply these methods to data where the underlying model must satisfy some structure. Overall, in each chapter, the intersection of optimization methods and statistics is highlighted. All algorithms proposed output the solution to some optimization problem, which we show have desirable properties both empirically and theoretically.In Chapter 2, we analyze biclustering in high dimensions. If data is viewed as a matrix, then biclustering in high dimensions requires structure with submatrices of similar magnitude and many columns of zero to filter out unhelpful features. A biconvex modification is made to convex biclustering to enable feature selection to be performed jointly with bicluster fitting. We propose two efficient algorithms: biconvex biclustering, a proximal alternating minimization scheme which weighs features and fits biclusters; and adaptive biconvex biclustering, which updates the affinity hyperparameters while fitting for faster convergence and increased performance. In addition to proving that biconvex biclustering converges to a critical point of the proposed objective function, we also prove a finite-sample bound on the mean squared error (in terms of the chosen features) of local optima. These bounds apply for a wide range of input affinity hyperparameters, helping explain the performance in practice. Extensive simulation studies show our algorithm outperforms peer methods in terms of both biclustering and feature selection. Additionally, we apply our method to a gene microarray dataset of lymphoma samples, recovering underlying data labels while giving additional interpretation to the mRNA samples (features) via the column groupings and fitted weights.In Chapter 3, we revisit the affinities common in convex clustering algorithms, such as the one proposed in Chapter 2. The affinities are a required input to convex clustering algorithms that dramatically affect the quality of the solutions. We expand on several results in the literature to help understand which properties of affinities affect underlying solution quality and the resulting structure of fitted clusters. In particular, we use the equivalence of affinities to graphs to prove rates of convergence for centroid recovery. A framework for this proof is built using properties of random walks and random graph models. Through the form of a finite-sample bound for mean squared error and additional empirical results, we argue proper tuning of hyperparameters to convex clustering problems should also include tuning of input affinity weights.In Chapter 4, we show how optimization can also be used to sample from constrained posterior distributions, giving uncertainty estimates for statistical problems that must satisfy some constraint. We prove the weighted Bayesian bootstrap, a method for approximate sampling of a posterior distribution, can be extended to sample general constrained posterior distributions under regularity conditions. The method entails a simple algorithm that can take advantage of fast tools from convex optimization. Under regularity conditions, we show the asymptotic distribution of samples from the constrained weighted Bayesian bootstrap has a covariance matching an efficient estimator, the restricted maximum likelihood estimator. We assess the method empirically on a variety of constrained Bayesian problems, demonstrating broad applicability of the method. The constrained weighted Bayesian bootstrap is able to quickly sample from constrained posteriors while providing adequate uncertainty quantification for problems solved via optimization methods that deliver only a point estimate. As a case study, using constraints required in European-style option prices, uncertainty estimates of an option pricing surface are derived with constrained weighted Bayesian bootstrap.