Multimodal Sensing and Algorithms to Monitor, Model, and Predict Human Interactions With the Built Environment

Abstract

Rapid urbanization is placing new demands on public infrastructure worldwide, creating challenges for cities in managing essential services like transit, public spaces, and emergency response. While "smart city" technologies are now commonplace, they often overlook complex human behavior and fail to predict how people interact with infrastructure systems. This dissertation addresses that gap through a people-centered approach by integrating novel sensing technologies with advanced predictive models to better understand and forecast human activity. In order to improve the utility of urban services for overall community wellbeing, the overarching goal of this research is to develop scalable sensing systems and algorithms that can monitor, model, and predict realistic human interactions with the built environment. The research is organized around three aims.The first aim develops multimodal sensing architectures that unify wearable and environmental sensors to monitor human-environment interactions in labor-intensive occupations. This work advances the idea of Cyber-Physical-Social Systems (CPSS) by integrating physiological and environmental monitoring to interpret human wellness in context. The technical foundation is a custom Bluetooth Low Energy system that synchronizes data from multiple wearable and environmental sensors. In a validation study with structural firefighters performing live-fire training, fusing both data modalities led to pronounced improvements in classifying operational phases and predicting self-reported wellness, including for new individuals not previously observed.The second aim addresses the privacy and scalability limits of video monitoring by developing deep learning methods to locate pedestrians non-intrusively using ground vibrations from their footsteps. This research introduces novel deep learning algorithms for estimating time difference of arrival (TDOA) that achieve sub-meter localization accuracy for pedestrians over large outdoor areas. A key contribution is an automated ground-truth generation process, where a temporarily deployed video camera and computer vision model label training data without manual input, enabling scalable deployment to new sites. The work is extended to multiple simultaneous pedestrians through a Bayesian particle filter based on the theory of random finite sets.The third aim tackles the limitations of traditional pedestrian simulations, which require large amounts of data and generalize poorly across different scenes. This research explores using generative video models as more flexible simulation tools. First, a benchmark framework (PEDRA) is introduced to evaluate how well current video diffusion models produce realistic pedestrian dynamics, judged against real-world data. Then, reinforcement learning is used to post-train a video diffusion model with a verifiable counting reward, improving the persistence of pedestrian agents across generated video frames. This approach demonstrates progress toward simulating complex pedestrian motion in new environments using generative models without site-specific data.The broader impacts of this research include enhanced planning and safety for urban infrastructure, improved wellness monitoring for emergency responders, and new tools for inclusive urban design. Overall, this dissertation provides sensing and simulation tools to help address urgent societal needs as cities continue to grow.