AI for Medical Imaging: Foundation Models, 2D-to-3D Reconstruction, and Clinical Applications

Abstract

Accurate and generalizable medical image segmentation remains a core challenge in clinical AI due to the diversity of imaging modalities, anatomical variability, and limited labeled data. Recent advances in vision foundation models, such as the Segment Anything Model (SAM), offer a new opportunity to build broadly applicable segmentation systems with minimal task-specific supervision. However, these models are pre-trained on natural images, and their zero-shot performance on medical imaging tasks remains limited. Adapting such models to diverse clinical domains—while maintaining generalization and interpretability—remains an open research problem.This dissertation explores strategies for developing and applying medical image segmentation algorithms, with a focus on adapting vision foundation models to the medical domain and integrating segmentation into downstream clinical and technical applications.In Chapter 2, we investigate multiple fine-tuning strategies—including task-specific supervised learning, prompt-based adaptation, and task-agnostic self-supervised pretraining—that substantially improve segmentation accuracy and generalization. These methods are evaluated across diverse datasets and in few-shot settings, offering practical pathways for adapting foundation models in real-world clinical contexts.Building on these findings, Chapter 3 introduces SegmentAnyBone, the first vision foundation model designed for musculoskeletal MRI. This model enables segmentation of any bone at any body location, demonstrating strong performance across anatomical regions, MRI sequences, and clinical variations.Recognizing that segmentation is often a stepping stone rather than the final goal in clinical practice, Chapter 4 presents two independent projects that apply segmentation-driven pipelines to support 2D-to-3D anatomical reconstruction. The first project generates high-resolution anatomical structures from multi-view MRIs by combining segmentation with deformable registration. The second project integrates segmentation with 3D geometric reasoning to reconstruct fracture angles in 3D space from biplanar X-rays.In Chapter 5, we explore how segmentation-derived features can more directly support clinical applications, particularly by enhancing surgical risk prediction. By combining image features extracted from abdominal CTs with standard clinical variables, we develop a more accurate risk prediction model that improves the prediction of postoperative mortality and morbidity beyond what existing models can achieve.Together, Chapters 4 and 5 highlight how segmentation models can be embedded into broader clinical workflows to support decision-making and enable more automated, precise assessments.In conclusion, this dissertation presents a comprehensive investigation into the development and deployment of vision foundation models for medical image segmentation, as well as the use of segmentation as a core component in both technical pipelines and clinical model development. It bridges algorithmic innovation with clinical relevance, contributing new tools, insights, and benchmarks toward building robust, generalizable, and impactful AI systems for healthcare.