Resource-efficient and Context-aware Edge Computing-supported Extended Reality Systems

Abstract

Extended reality (XR), including virtual reality (VR) and augmented reality (AR), has been showing promises in many applications including gaming, education, and healthcare. Immersive XR systems require context awareness of the user and environmental conditions. High expectation for XR, coupled with their known resource-hungry nature, requires optimizations of XR systems to reduce their communication and computing resource consumption. In this dissertation, we focus on characterizing user and environmental contexts and building resource-efficient and adaptive systems to respond to these characteristics. First, we model and exploit patterns of one aspect of user behaviors in VR systems, that is, VR viewport pose that represents the location and the orientation of VR devices. Then, we delve into enhancing the accuracy and resource efficiency of a crucial component for XR user and environmental context awareness: simultaneous localization and mapping (SLAM) that performs device pose tracking and environmental mapping concurrently. Our approach involves proposing an SLAM system that intelligently adapts to both communication and computation resource constraints of XR devices to maintain high SLAM performance. We also lessen the computational loads on XR devices and enhance the context awareness capabilities by using external Internet of things (IoT) cameras.We first present our work on analyzing and exploiting pose characteristics for virtual content generation. We develop the first statistical model of viewport poses in VR. Our statistical model of users’ VR viewport pose comprises the models of pose components, orientation and position, based on the experimental measurements. Under the developed VR viewport pose model, we derive analytical results for the pixel similarity between different VR frames. Based on our analytical model, we adaptively determine which contents are reused across different VR frames. We evaluate the developed algorithms by implementing them on Meta Quest 2-based edge-assisted VR systems, and demonstrate the systems running in real time, supporting the full VR frame rate, and outperforming baselines on measures of frame quality and bandwidth consumption.We then present our work on performing adaptive and robust SLAM under resource constraints for AR devices. We develop the first uncertainty quantification model for pose estimation in visual (V-) and visual-inertial (VI-) SLAM under edge computing-based architectures. We apply the developed uncertainty quantification model to efficiently and optimally select subsets of frames to build pose graphs under limited computation and communication resources. We implement the design in conjunction with the state-of-the-art V- and VI-SLAM framework, and show that it reduces the pose estimation error under constrained bandwidth. This is the first theoretically grounded work that uses optimization-based algorithms to identify and transmit the most informative frames, as opposed to existing works that rely on heuristics in making offloading decisions.Finally, we propose enhancing XR systems’ SLAM performance with external IoT cameras. We integrate IoT data into all primary modules of the SLAM systems, designing both tight and loose fusion methods to integrate with the visual and inertial measurements from XR devices. In tight integration, additional terms related to IoT information are incorporated in the map optimization and pose optimization problems. For loose integration, IoT data aids in tasks including pose initialization, pose rectification, and loop detection. Our experimental results, conducted using a Unity game engine-based emulator, demonstrate that our IoT-enhanced SLAM system, IoTSLAM, outperforms ORB-SLAM3 and the loose fusion approach by 25.5% and 16.1% across all sequences in both V- and VI-configurations.