Statistical Analysis of Tag Semantics, Classification, and Genre Evolution of Movies to Improve Recommendation Systems

Abstract

This thesis analyzes the MovieLens 20M dataset and its tag genome extension to un-derstand movie semantics, classification, and genre evolution. We address three main objectives. First, we simplify 1,128 correlated tag variables into interpretable dimen- sions via tag merging and Principal Component Analysis (PCA), revealing four mean- ingful semantic axes: critically acclaimed films, big-budget blockbusters, cult/experimental cinema, and stylized crime thrillers. Second, we classify cult classic films using LASSO logistic regression and Classification and Regression Trees (CART), iden- tifying interpretable tag-based decision rules despite the inherent subjectivity of the label. Third, we analyze genre co-occurrence patterns through network visualization and hierarchical clustering, and model the evolution of genre market shares from 1941 to 2015 using a Bayesian Multivariate Dynamic Linear Model (MVDLM) applied to centered log-ratio transformed compositional data. Results show that the film indus- try transitioned from an expansionary phase to a zero-sum competitive equilibrium by the 2000s, with genre correlations shifting substantially over time. Together, these findings demonstrate the value of interpretable statistical methods for understanding large-scale movie data.