Practical Solutions to Neural Architecture Search on Applied Machine Learning

The advent of Artificial Intelligence (AI) propels the real world into a new era characterized by remarkable design innovations and groundbreaking design automation, primarily fueled by Deep Neural Networks (DNN). At the heart of this transformation is the progress in Automated Machine Learning (AutoML), notably Neural Architecture Search (NAS). NAS lays a robust foundation for developing algorithms capable of automating design processes to determine the optimal architecture for academic benchmarks. However, the real challenge emerges when adapting NAS for Applied Machine Learning (AML) scenarios: navigating the complex terrain of design space exploration and exploitation. This complexity arises due to the heterogeneity of data and architectures required by real-world AML problems, an aspect that traditional NAS approaches struggle to address fully.

To bridge this gap, our research emphasizes creating a flexible search space that reduces reliance on human-derived architectural assumptions. We introduce innovative techniques aimed at refining search algorithms to accommodate greater flexibility. By carefully examining and enhancing search spaces and methodologies, we empower NAS solutions to cater to practical AML problems. This enables the exploration of broader search spaces, better performance potential, and lower search process costs.

We start by challenging homogeneous search space design for multi-modality 3D representations, proposing ``PIDS'' to enable joint dimension and interaction search for 3D point cloud segmentation. We consider two axes on adapting point cloud operators toward multi-modality data with density, geometry, and order varieties, achieving significant mIOU improvement on segmentation benchmarks over the state-of-the-art 3D models.To implement our approach efficiently in recommendation systems, we develop ``NASRec'' to support heterogeneous building operators and propose practical solutions to improve the quality of NAS on Click-Through Rates (CTR) prediction. We propose an end-to-end full architecture search with minimal human priors. We provide practical solutions to tackle scalability and heterogeneity challenges in NAS, outperforming manually designed models and existing NAS models on various CTR benchmarks. Finally, we pioneer our effort on industry-scale CTR benchmarks and propose DistDNAS to optimize search and serving efficiency, producing smaller and better recommendation models on a large-scale CTR benchmark. Intuited by the discoveries in NAS, we additionally uncover the underlying theoretical foundations of residual learning on computer vision foundation research and envision the prospects of our research on Artificial Intelligence, including Large Language Models, Generative AI, and beyond.