From Concepts to Efficiency: Advancing Interpretable and Scalable Machine Learning Systems

Abstract

Machine learning systems have recently proven to be powerful across a wide range of applications, from everyday tasks to highly specialized domains such as scientific discovery. However, two major challenges hinder their widespread adoption: (1) the lack of interpretability in their reasoning processes, which undermines their reliability in high-stakes applications, and (2) the lack of scalability, which impedes the democratization of deployment. Many existing machine learning algorithms suffer from one or both of these issues, which can often be interrelated. This thesis addresses these fundamental challenges in two areas: dimensionality reduction (DR) models and neural networks.First, we examine dimensionality reduction techniques, which play a crucial role in identifying and extracting meaningful concepts from high-dimensional data. We establish key design principles for preserving both local and global structures, ensuring that the extracted concepts are reliable. Based on these principles, we introduce PaCMAP, a novel DR algorithm that outperforms existing methods in terms of structure preservation, speed and scalability. Additionally, we conduct a comprehensive benchmarking study to evaluate widely used DR techniques. To further accommodate large-scale datasets and online learning scenarios, we develop ParamRepulsor, a parametric DR method incorporating hard negative mining, achieving state-of-the-art performance.Next, we address the challenge of interpretability and efficiency in neural networks. In visual models, reasoning through concepts has been widely accepted as a means of enhancing interpretability, but existing concept discovery methods often require human supervision, making them prone to bias and errors. We propose SegDiscover, an unsupervised framework for identifying semantically meaningful visual concepts in complex imagery. SegDiscover surpasses previous approaches on challenging datasets such as Cityscapes and COCO-Stuff, providing a robust tool for improving model interpretability.For large language models (LLMs), we investigate Mixture-of-Experts (MoE) architectures, which offer greater interpretability than standard dense models but often suffer from inefficiencies during deployment. To address these inefficiencies, we propose techniques such as dynamic gating, expert buffering, and load balancing, significantly improving throughput and reducing memory usage. These optimizations make MoE models more scalable and practical for real-world applications.