Harnessing Multimodal Data with Deep Learning in Radiology: Dataset Curation Strategies and Clinical Applications

Abstract

Thanks to the development of modern data-driven technologies in the healthcare domain, such as institutional Picture Archiving and Communication System (PACS) and data warehouses, large volumes of electronic health record (EHR) data in different modalities are readily available for data-driven research in radiology. Although the proliferation of data has enriched the context of addressing clinical problems, the complex relationships among the multimodal data sources, with their vast potential to advance healthcare research, have remained underexplored. We hypothesize that leveraging the intrinsic connections across multimodal data can be crucial to answering the clinical questions in radiology.This dissertation highlights our work in bridging this knowledge gap through two main topics. First, we will illustrate curation methodologies that transform the raw, heterogeneous, and multimodal data into a well-curated form specifically tailored for each downstream research objective. Second, we will demonstrate the practical and unique values of the well-curated datasets by addressing various problem statements in radiology. Specifically, we will focus on illustrating the methodologies applied to three distinct domains in radiology, including thyroid radiology, musculoskeletal (MSK) radiology, with a specialty in knee radiographs, and neuroradiology, with focuses on brain and cervical spine MRI. For thyroid radiology, we demonstrated a Multistep Automated Data Labeling Procedure (MADLaP) to curate proper datasets on ultrasound for research in thyroid nodule cancer. For MSK radiology, we presented an automated labeling approach for knee radiographs that can improve the image classification task of identifying knee abnormalities. For neuroradiology, first, we conducted a longitudinal study that utilized deep-learning-based volumetric analysis to investigate the progression of brain tumor on MRI. We designed and evaluated a framework that automates the radiologists’ response assessments on patients with glioblastoma. Second, we presented the Duke University Cervical Spine MRI Segmentation Dataset (CSpineSeg). In this dataset, we provided open-source MRI data, annotations on vertebral bodies and intervertebral discs, and structured data to facilitate cervical spine disease research. To summarize, this dissertation explores a systematic approach to leverage connections across multimodal data in radiology and evaluates the utility of well-curated datasets in downstream clinical research.