Towards Efficient and Robust Deep Neural Network Models

Recently, deep neural network (DNN) models have shown beyond-human performance in multiple tasks. However, DNN models still exhibit outstanding issues on efficiency and robustness that hinder their applications in the real world. For efficiency, modern DNN architectures often contain millions of parameters and require billions of operations to process a single input, making it hard to deploy these models on mobile and edge devices. For robustness, recent research on adversarial attack shows that most DNN models can be misled by tiny perturbations added on the input, leaving doubts on the robustness of DNNs in security-related tasks. To tackle these challenges, this dissertation aims to advance and incorporate techniques from both fields of DNN efficiency and robustness, leading towards efficient and robust DNN models.

My research first advances model compression techniques including pruning, low-rank decomposition, and quantization to push the boundary of efficiency-accuracy tradeoff in DNN models. For pruning, I propose DeepHoyer, a new sparsity-inducing regularizer that is both scale-invariant and differentiable. For decomposition, I apply the sparsity-inducing regularizer on the decomposed singular values of DNN layers, together with an orthogonality regularization on the singular vectors. For quantization, I propose BSQ to achieve optimal mixed-precision quantization scheme by exploring bit-level sparsity, mitigating the costly search through the large design space of quantization precision. All these works successfully achieve DNN models that are both more accurate and more efficient than state-of-the-art methods. For robustness improvement, I change the previously undesired accuracy-robustness tradeoff of a single DNN model into an efficiency-robustness tradeoff of a DNN ensemble, without hurting the clean accuracy. The method, DVERGE, combines a vulnerability diversification objective and previously investigated model compression techniques, leading to an efficient ensemble whose robustness increases with the number of sub-models. Finally, I propose to unify the pursuit of accuracy and efficiency as an optimization towards robustness against weight perturbation. Thus, I introduce Hessian-Enhanced Robust Optimization to achieve highly accurate model that are robust to post-training quantization. The accomplish of my dissertation research paves way towards controlling the tradeoff between accuracy, efficiency and robustness, and leads to efficient and robust DNN models.