Towards Real Time 3D Reconstruction and Display of Synchronized Fourier Lightfield Video for Novel Microsurgical Applications

Abstract

Although state-of-the-art stereo microscopes can provide surgeons with qualitative 3D information, obtaining quantitative 3D scene information serves an increasingly active role in high-risk settings such as robotic or automated microsurgery, where precise 3D tooltracking and tissue deformation mapping are critical. This thesis extends previous work on the Fourier Lightfield Multiview Stereoscope (FiLMScope) with a novel, end-to-end firmware and software pipeline that processes a high-throughput datastream to display high-resolution, colored 3D reconstructions in real time (>4Hz). We introduce a multicamera array capable of capturing up to 32 snapshots from multiple angular perspectives per frame, each at video rate with >4K resolution. We also describe new insights, such as inter-frame bootstrapping and system hyperparameter optimization, and their potential limitations in highly dynamic scenes. In addition, we incorporate code optimization techniques and parallel processing, which enables quick routing and display of overlaid 3D imagery within a graphical user interface, thus facilitating real-time clinician interaction. Finally, we demonstrate the utility of the pipeline in various imaging phantoms and realistic surgical settings.