Improving Deep Learning with Dynamical System

Abstract

This dissertation addresses the fundamental limitations of traditional deep learning architectures in capturing complex temporal dynamics and handling high-dimensional data efficiently. Despite their success, current neural network models often struggle with continuous-time representations, long-range dependencies, and computational scalability when processing sequential or time-varying phenomena.To overcome these challenges, we develop a unified framework that integrates dynamical systems theory with neural network architectures. Our methodology enhances the expressivity and efficiency of existing deep learning systems by combining principles from differential equations and continuous-time modeling. We first consider the case where we have temporal data from an unidentified system and aim to model the dynamics that the data follows. Neural Ordinary Differential Equations (NODEs) model temporal data with continuous and expressive dynamics while keeping training costs in check. However, continuous ODE dynamics fail to model even simple toy example datasets. We extend NODEs to Characteristic-Neural Ordinary Differential Equations (C-NODEs) to model the temporal data with Hyperbolic Partial Differential Equations (Hyperbolic PDEs) while still paying the training prices of NODEs. Our empirical results on C-NODEs demonstrate significant improvements across multiple domains over NODEs while using similar training and inference budgets.Secondly, we consider the problem of traditional numerical PDE solvers, where we have the explicit algebraic form of a parametric family of high-dimensional phenomena, but only have limited data in comparison to the dimensionality of the dynamics. In particular, we consider Monte-Carlo solvers approximating point-wise solutions to high-dimensional PDEs. While Monte-Carlo solvers are elegant, they suffer from a sequential computation bottleneck, and modern computer hardware like the GPU can not provide substantial accelerations. We apply Girsanov's theorem and implement meta-learning approaches to expedite the solver by reusing previous simulation results and parallelizing the computations. Empirical results show the computational time reductions while having low approximation errors.Last but not least, we consider the case where we have the algebraic form of the parametric family of an up to three-dimensional PDE and some data, and would like to approximate the functional form of the solutions. While existing works rely primarily on empirical training, overlooking the inherent mathematical relationships between different PDEs that could inform their solutions, we develop a numerical-neural network hybrid PDE solvers that leverage these connections to achieve state-of-the-art trade-off efficiencies between computing time and approximation accuracy. These findings establish that incorporating dynamical systems principles into deep learning architectures can substantially enhance model performance, computational efficiency, and theoretical understanding across diverse application domains, opening new avenues for scalable artificial intelligence systems.