Browsing by Author "Carlson, David"

Deep representation learning has shown its effectiveness in many tasks such as text classification and image processing. Many researches have been done to directly improve the representation quality. However, how to improve the representation quality by cooperating ancillary data source or by interacting with other representations is still not fully explored. Also, using representation learning to help other tasks is worth further exploration.

In this work, we explore these directions by solving various problems in natural language processing and image processing. In the natural language processing part, we first discuss how to introduce alternative representations to improve the original representation quality and hence boost the model performance. We then discuss a text representation matching algorithm. By introducing such matching algorithm, we can better align different text representations in text generation models and hence improve the generation qualities.

For the image processing part, we consider a real-world air condition prediction problem: ground-level $PM_{2.5}$ estimation. To solve this problem, we introduce a joint model to improve image representation learning by incorporating image encoder with ancillary data source and random forest model. We the further extend this model with ranking information for semi-supervised learning setup. The semi-supervised model can then utilize low-cost sensors for $PM_{2.5}$ estimation.

Finally, we introduce a recurrent kernel machine concept to explain the representation interaction mechanism within time-dependent neural network models and hence unified a variety of algorithms into a generalized framework.

Effective decision making requires understanding the uncertainty inherent in a problem. This covers a wide scope in statistics, from deriving an estimator to training a predictive model. In this thesis, I will spend three chapters discussing new uncertainty methods developed for solving individual and population level inference problems with their theory and applications in causal inference. I will also detail the limitations of existing approaches and why my proposed methods lead to better performance.

In the first chapter, I will introduce a novel approach, Collaborating Networks (CN), to capture predictive distributions in regression. It defines two neural networks with two distinct loss functions to approximate the cumulative distribution function and its inverse respectively and collectively. This gives CN extra flexibility through bypassing the necessity of assuming an explicit distribution family like Gaussian. Empirically, CN generates sharp intervals with reliable coverage.

In the second chapter, I extend CN to estimate the individual treatment effect in observational studies. It is augmented by a new adjustment scheme developed through representation learning, which is shown to effectively alleviate the imbalance between treatment groups. Moreover, a new evaluation criterion is suggested by combing the estimated uncertainty and variation in utility functions (e.g., variability in risk tolerance) for more comprehensive decision making, while traditional approaches only study an individual’s outcome change due to a potential treatment.

In the last chapter, I will present an analysis pipeline for causal inference with propensity score weighting. Comparing to other pipelines for similar purposes, this package comprises a wider range of functionalities to provide an exhaustive design and analysis platform that enables users to construct different estimators and assess their uncertainties. Itoffers six major advantages: it incorporates (i) visualization and diagnostic tools of checking covariate overlap and balance, (ii) a general class of balancing weights, (iii) comparison for multiple treatments, (iv) simple and augmented (doubly-robust) weighting estimators, (iv) nuisance-adjusted sandwich variances, and (v) ratio estimands for binary and count outcomes.

This thesis presents novel methods for processing electrophysiological time-series from simultaneously recorded electrodes in a brain, as well as providing new inference techniques that are more generally applicable. On spike sorting, I introduce Bayesian nonparametric methods to process multiple electrodes simultaneously, which improves performance when the electrode spacing is less than 100 microns. Furthermore, by treating the spike sorting problem as a single deconvolutional model instead of the conventional 2-step procedure with detection and clustering steps, the over- lapping spike problem is ameliorated. I then show that these detected neurons and their spike trains have dynamic relationships with local field potentials in distinct brain regions, and that the number of distinct relationships appears to cluster.

While these models approach an important scientific problem, it is necessary to have efficient inference in computationally-intensive models. To this end, I intro- duce novel methods for Variational Bayesian inference, as well as introducing a new stochastic inference algorithm called "Stochastic Spectral Descent," which mimics Stochastic Gradient Descent but operates in the Shatten-infinity norm. I show that several common machine learning problems naturally operate in the Shatten-infinity norm, and that this descent method mimics the natural geometry and greatly improves learning efficiency.