Advancing Regularization Methods for Interpretable and Robust Deep Learning

Abstract

Over the past decade, the trend of deep learning has made machine learning (ML) models significantly larger, deeper, and more complex. Increases in model complexity and innovations in scaleable learning algorithms have yielded rapid gains in downstream task performance. However, the near exponential growth in model complexity incurs numerous costs which limit the models' utility for real-world decision systems. We focus on two issues - the uninterpretable and brittle reasoning process of large ML models - and propose advanced regularization methods to address them. Between two threads of research, we investigate regularization methods which can (1) reliably clarify the structure of representations learned by ML models and (2) provide a more favorable tradeoff between adversarial robustness and task performance.In our first thread, we study regularization methods which reliably clarify the structure of representations captured by large ML models. Starting with the unsupervised disentanglement of learning representations, we introduce theoretically-grounded inductive biases which elucidate the global structure of the learned representation. Turning our attention to the local structure of learned representations, we introduce a method to reveal the topological dimension of data captured by diffusion models. This theoretical understanding of the local structure captured by diffusion models provides valuable insights on the robustness-performance tradeoff in the second thread.In our second thread, we identify theoretically-grounded pathways in which regularization for robustness compromises the ML model's performance on the original task (i.e., the robustness performance tradeoff). We analyze this conflict directly from a parameter optimization perspective and introduce a from of complex-valued neural network which alleviates this tradeoff. Next, we extract the underlying structure of the data distribution using diffusion models (from the first thread) to reveal that current adversarial training methods violate this structure, leading to unnecessary performance loss. Last, we argue that local structural knowledge should be leveraged to refine our assumptions on adversarial examples and to yield a more favorable tradeoff between robustness and performance.