Rethinking Dropout in Transformer-Based Neural Operators: Mechanisms, Effects, and Design Principles

Abstract

Transformer models have achieved significant success in natural language processing and computer vision and are now increasingly applied in scientific machine learning to solve complex partial differential equations. However, conventional training techniques, including dropout regularization, are frequently adopted from related fields without a critical evaluation of their appropriateness for partial differential equation (PDE) operator learning. This thesis investigates the effectiveness of dropout in transformer-based operator learning for elliptic PDEs through systematic experiments. Additionally, we introduce a correlated dropout framework to investigate the influence of spatial-frequency characteristics of dropout masks on results, comparing independent, low-frequency, high-frequency, and local Gaussian correlated dropout methods. Our findings consistently show that removing dropout improves accuracy in transformer-based elliptic PDE operator learning. Based on these results, we suggest removing dropout as a practical guideline for designing transformer-based operator learning models.