Posterior Degeneracy and Prior Sensitivity in Bayesian Topic Models

Abstract

Topic modeling is a widely used approach for exploring large collections of documents byautomatically discovering recurring themes (``topics'') and summarizing each document by how strongly it expresses those themes. Bayesian topic models such as Latent Dirichlet Allocation and related nonparametric extensions provide a probabilistic framework for this task, but their behavior can depend strongly on prior specification. This dependence is especially pronounced when the vocabulary (the set of distinct word types) is very large while individual documents are short. In this high-dimensional and sparse regime, the data may provide limited information about how words should be grouped into topics, leading to unstable or poorly interpretable inferred structure.This thesis studies prior sensitivity in Bayesian topic modeling from a vocabulary-growthperspective. In particular, it investigates how Dirichlet-based prior specifications scale with vocabulary size and how different scaling regimes affect posterior inference on latent topic structure. The thesis has four main goals. First, it develops theoretical results that characterize how posterior learning changes with vocabulary size under different prior-scaling choices. Second, it translates these theoretical regimes into observable diagnostics of instability during inference. Third, it validates the theoretical predictions through exact calculations, simulations, and corpus-based experiments that vary vocabulary size through preprocessing and controlled prior settings. Finally, it extends the framework to partially exchangeable grouped corpora using dependent Bayesian nonparametric priors, with emphasis on multivariate species sampling models that make cross-group information sharing explicit through tie behavior.Overall, the thesis clarifies when and why Dirichlet-based topic models become sensitive toprior specification under vocabulary growth and characterizes the regimes in which posterior inference degenerates.