From Adversaries to Anomalies: Addressing Real-World Vulnerabilities of Deep Learning-based Vision Models

Abstract

Deep Neural Networks (DNNs) have driven the performance of computer vision to new heights, which has led to them to being rapidly integrated into many of our real-world systems. Meanwhile, the majority of research on DNNs remains focused on enhancing accuracy and efficiency. Furthermore, the evaluation protocols used to quantify performance generally assume idealistic operating conditions that do not well-emulate realistic environments. For example, modern benchmarks typically have balanced class distributions, ample training data, consistent object scale, minimal noise, and only test on inputs that lie within the training distribution. As a result, we are currently integrating these naive and under-tested models into our trusted systems! In this work, we focus on the robustness of DNN-based vision models, seeking to understand their vulnerabilities to non-ideal deployment data. The rallying cry of our research is that before these models are deployed into our safety-critical applications (e.g., autonomous vehicles, defense technologies), we must attempt to anticipate, understand, and address all possible vulnerabilities. We begin by investigating a class of malignant inputs that are specifically designed to fool DNN models. We conduct this investigation by taking on the perspective of an adversary who wishes to attack a pretrained DNN by adding (nearly) imperceptible noise to a benign input to fool a downstream model. While most adversarial literature focuses on image classifiers, we seek to understand the feasibility of attacks on other tasks such as video recognition models and deep reinforcement learning agents. Sticking to the theme of \textit{realistic} vulnerabilities, we primarily focus on black-box attacks in which the adversary does not assume knowledge of the target model's architecture and parameters. Our novel attack algorithms achieve surprisingly strong effectiveness, thus uncovering new serious potential security risks.While malignant adversarial inputs represent a critical vulnerability, they are still a fairly niche issue in the context of all problematic inputs for a DNN. In the second phase of our work, we turn our attention to the open-set vulnerability. Here, we acknowledge that during deployment, models may encounter novel classes from outside of their training distribution. Again, the majority of works in this area only consider image classifiers for their simplicity. This motivates us to study the more complex and practically useful open-set object detection problem. We address this problem in two phases. First, we create a tunable class-agnostic object proposal network that can be easily adapted to suit a variety of open-set applications. Next, we define a new Open-Set Object Detection and Discovery (OSODD) task that emphasizes both known and unknown object detection with class-wise separation. We then devise a novel framework that combines our tunable proposal network with a powerful transformer-based foundational model, which achieves state-of-the-art performance on this challenging task.We conclude with a feasibility study of inference-time dynamic Convolutional Neural Networks (CNNs). We argue that this may be an exciting potential solution for improving robustness to natural variations such as changing object scale, aspect ratio, and surrounding contextual information. Our preliminary results indicate that different inputs have a strong preference for different convolutional kernel configurations. We show that by allowing just four layers of common off-the-shelf CNN models to have dynamic convolutional stride, dilation, and size, we can achieve remarkably high levels of accuracy on classification tasks.