Interpretable and Trustworthy AI Systems for Healthcare

Abstract

Artificial intelligence (AI) has demonstrated strong predictive performance across a range of clinical tasks, including mortality prediction. However, in high-stakes domains like healthcare, predictive performance alone is not enough — failures carry direct consequences for patients. Medical experts must be able to review and understand model predictions and ensure regulatory compliance, which requires models that are inherently interpretable and controllable. AI in healthcare extends beyond predictive models. With advances in large language models, LLM-based agents hold substantial potential for healthcare tasks. However, healthcare data is sensitive and any leakage or corruption has a direct impact on patients, and AI systems must comply with regulations such as HIPAA and GDPR. Therefore, we must ensure LLM agents handle data securely, remain auditable, and avoid unregulated access to sensitive data. In this dissertation, we address three specific problems in healthcare AI: how simpler traditional machine learning methods compare to complex AI models for EHR time series prediction, how to build inherently interpretable architectures for clinical text classification, and how to enable secure LLM agents for sensitive data analysis.We examine whether, and under what conditions, transformer models offer performance advantages over classical machine learning methods for EHR time series classification. Using data from over 500,000 patients receiving maintenance hemodialysis, we compare classical ML models --- logistic regression, Gradient Boosting Machines (GBM), and Extreme Gradient Boosting (XGBoost) --- to transformer-based models on a one-year mortality prediction task. We systematically vary training dataset size (from 500 samples to the full training set), feature representation --- ranging from last-visit values only, to summary statistics computed over all visits, to summary statistics computed over monthly windows --- and the length of patient history used as input, spanning one, three, and six months of patient records. We find that more granular feature representations tends to yield better performance, and that longer patient history as input consistently improves performance, particularly for transformers. On the full training dataset using six months of patient history with monthly summary statistics as features, XGBoost achieves AUC-ROC of 0.8598, marginally outperforming transformers at 0.8568 and GBM at 0.8521. At 50,000 training samples, GBM achieves 0.8377 while transformers achieve 0.8305. Our experiments show that classical machine learning methods are competitive with, and sometimes outperform, transformer models, establishing them as strong baselines for EHR time series classification.Tabular and time series EHR data vary significantly across institutions --- different institutions collect different features, use different units, and follow different conventions, making models difficult to transfer across settings. Moreover, structured fields often fail to capture nuanced clinical observations made by physicians and nurses. Clinical text, in contrast, uses natural language that generalizes better across institutions and captures richer clinical information. To build predictive models from clinical text, we first need to de-identify protected health information (PHI) such as patient names, dates, and medical record numbers to protect patient privacy. NER models serve this purpose. Beyond de-identification, NER models can also extract clinical entities such as medications, diagnoses, and procedures that can potentially serve as features for interpretable predictive models. However, features evolve, and to modify NER models to recognize new entities, we need to re-annotate and re-train, which is expensive. To mitigate this, we propose the Partial Label Model (PLM), a method for training a unified NER model using multiple partially annotated datasets, where each dataset is annotated with a different set of entity types. We evaluate PLM on six diverse NER datasets across various taxonomy expansion scenarios. Our experiments show that PLM outperforms cross-annotation and continual learning baselines by 0.5--2.5 F1 points. Further, in data-scarce scenarios, PLM significantly outperforms other baselines.While traditional NER models require supervised training data with predefined entity types, large language models can recognize entities from natural language descriptions or a few examples without requiring annotated datasets. Leveraging the capabilities of LLMs, we develop ClinStructor, a three-stage pipeline that generates task-relevant questions, extracts answers to these questions from clinical notes, and fine-tunes an LLM on the resulting structured question-answer representations. This approach enables explicit control over which attributes are available to the model --- if the question set includes ``What medications is the patient currently taking?'', we know the model has access to medication information when making predictions. On ICU mortality prediction using MIMIC-III admission notes with over 7,000 training examples, ClinStructor achieves AUC-ROC of 0.860 compared to 0.882 for black-box models trained directly on raw text. While ClinStructor provides control over which attributes are used, it does not explain how those attributes influence the model's predictions. We therefore introduce CALM (Classification with Additive Large Language Models), an inherently interpretable architecture that works on semi-structured textual input, such as question-answer pairs from ClinStructor. CALM processes each textual component independently and combines their contributions additively to produce a final prediction, making the contribution of each textual component explicit as part of the forward pass. We also propose CALM-Distill and CALM$^2$, extensions of CALM. We evaluate CALM and its extensions on three ICU mortality prediction datasets and seven LLM backbones ranging from 0.5B to 8B parameters, and show that CALM, especially with its extensions, performs comparable to black-box fine-tuning baselines while providing faithful local and global interpretability.Beyond predictive models, recent advances in large language models have enabled LLM-based agents with substantial potential for healthcare tasks---a physician could ask ``When was the last time this patient was prescribed medication X?'' and let the LLM agent perform the data analysis and answer it. However, healthcare data is sensitive, and any leakage or corruption has a direct impact on patients; AI systems must comply with regulations such as HIPAA and GDPR. We propose a security-first, tool-only framework that restricts LLM agents from accessing raw data or generating executable code, requiring them to interact with data exclusively through predefined, vetted tools. To answer complex queries, LLM agents must reason and invoke several tools sequentially. However, existing tool-calling benchmarks primarily evaluate simple tool selection tasks or focus on retrieving relevant tools from large collections, rather than assessing long sequential tool usage. We therefore introduce InData, a benchmark specifically designed to evaluate compositional, multi-step tool-based reasoning. InData consist of 100 tools, 114 tabular datasets, and 2,063 data analysis questions spanning three difficulty levels. We benchmark 15 open-source LLMs and find that performance degrades substantially as reasoning complexity increases --- for example, gpt-oss-120B attains 97.3\% accuracy on Easy questions but only 69.6\% on Hard questions, while Qwen3-1.7B drops from 65.5\% to 7.5\%, revealing a substantial gap in current LLMs' multi-step tool-based reasoning capabilities. To improve LLM reasoning ability, we further investigate test-time compute scaling, specifically Monte Carlo Tree Search with self-refinement (MCTSr). When ground-truth rewards are available, MCTSr yields substantial gains---for example, Llama 3 Instruct 70B improves from 81.3\% to 95.8\% accuracy on GSM8K. However, when ground-truth rewards are unavailable and we rely on LLM-based feedback, performance often falls below the no-search baseline. We also experimented using the ToolTalk dataset, and observed that MCTSr again degrades performance compared to the no-search baseline; for instance, Sonnet 3's F1 score drops from 0.706 to 0.559. We further analyze failure modes and find that, when LLM agents are asked to perform a task with insufficient information, instead of seeking clarification, they frequently hallucinate tool parameters --- for example, inventing usernames such as ``newuser'' or passwords such as ``password123'' when credentials are not provided and the LLM-based feedback model fails to catch such errors. To mitigate hallucination, we introduce a dedicated hallucination-detection module that checks if each tool parameter is specified by the user or reasonably inferred based on the context. While incorporating this module reduces hallucination, it results in over-cautious behavior, where the LLM asks the user to confirm even information provided earlier. Our analysis shows that self-reward and self-feedback models may not reliably improve LLM reasoning ability.Finally, we develop the Science Consultant Agent, an LLM-based agent that democratizes AI development for non-AI experts. The agent first collects task descriptions and requirements through structured questionnaires, then analyzes the responses, searches arXiv for relevant literature, and recommends appropriate modeling strategies along with supporting references and justifications. If provided with the required data, it can also automatically build standard ML baselines. We conducted internal user studies at Amazon Pharmacy, where 82\% of participants rated the overall experience as excellent or good and the rest satisfactory, all participants found the justifications convincing (73\% moderately to very convincing and the rest somewhat convincing), and all found the recommendations aligned with their expectations (63\% moderately to perfectly aligned and the rest somewhat aligned).This dissertation is a step toward AI systems that are accurate, interpretable, auditable, and secure --- properties essential for application in high-stakes domains like healthcare.