Advancing Multimodal Generative Models via Structured Representation Learning and Semantic Planning

Abstract

Modern multimodal generative models produce visually stunning images and videos, yet they frequently fail to follow complex instructions, maintain semantic coherence over long temporal horizons, or satisfy user-specified constraints such as privacy. These shortcomings stem from a common architectural bottleneck: monolithic end-to-end generators conflate high-level semantic understanding with low-level pixel synthesis, offering no explicit mechanism for abstraction, hierarchical reasoning, or constraint enforcement.This dissertation demonstrates that structured intermediate representations and explicit semantic planning can close this gap. We pursue two complementary thrusts---learning representations whose internal organization reflects semantic structure, and delegating high-level reasoning to dedicated planning modules---and develop them across five interconnected works that trace a progression from foundational tokenization to planning-guided, constraint-aware generation.We begin by showing that discrete video tokens, generated autoregressively with a three-dimensional sparse attention mechanism and pretrained on over 136 million text--video pairs, enable one of the first open-domain text-to-video models (GODIVA). We then reveal that the ordering within such discrete codes matters: Progressive Quantization VAE (PQ-VAE) learns a hierarchy of tokens ranked by semantic importance, where the leading codes capture global structure and subsequent codes add fine-grained detail, yielding superior compression and interpretability. Extending the principle of structured representation to continuous embeddings, we adapt Orthogonal Low-rank Embedding for self-supervised learning (SSOLE), overcoming collapse and sign-ambiguity barriers to achieve competitive performance on ImageNet without large batches, memory banks, or dual encoders.Building on these representation foundations, we introduce Plan-X, a framework that decouples semantic planning from visual synthesis: a multimodal language model produces structured spatio-temporal semantic tokens that guide a video Diffusion Transformer, dramatically reducing visual hallucination and improving instruction alignment. Finally, we demonstrate that the same representation-learning principles enable constraint-driven generation for privacy: an ID-augmented contrastive encoder, trained with the SSOLE objective on synthetic face-swap variants, learns identity-agnostic attribute codes that condition a diffusion generator to produce anonymized faces preserving expression, gaze, and pose while effectively erasing identity.Extensive experiments across video generation, image reconstruction, self-supervised learning, instruction-following synthesis, and privacy-preserving face anonymization confirm the central thesis: structured intermediate representations and explicit planning are essential for advancing multimodal generative models from visually plausible output toward faithful, controllable, and responsible generation.