Real-time Target Tracking in Fluoroscopy Imaging using Unet with Convolutional LSTM

Target localization precision is crucial for the treatment outcome of radiation therapy. In lung stereotatic body radiation therapy (SBRT), verifying target motion in the real time 2D fluoro images is often used as a vital tool to ensure adequate coverage of the target volume before the treatment delivery starts. However, accurate target localization in 2D fluoroscopy images is very challenging due to the overlapping anatomical structures in the projection images. The localization is often visually performed by physicians and physicists, which is a subjective process that depends on the experience of the clinician. In this paper, we have developed a deep learning network for automatic target localization to improve the efficiency and robustness of the process. Specifically, the deep learning network adopts a Unet architecture with a coarse-to-fine structure. In addition, we innovatively incorporate convolutional Long Short-Term Memory (LSTM) layer into the network to utilize the time correlation between the fluoro images. A Generative Adversarial method was used to train the network to further improve its localization accuracy. A hybrid loss was used to improve the feature learning during the training. The model was tested on a large amount of data generated by the digital X-CAT phantom. Various patient sizes, respiratory amplitudes, and tumor sizes and locations were simulated in the X-CAT phantoms to test the accuracy and robustness of the method. Our model has been proved with great accuracy not only on massive samples but also on specific set of samples. On massive samples, our model achieves IOU 0.92 and centroid of mass difference 0.16 and 0.07 cm in vertical and horizontal direction. On unique set of samples, the IOU is even higher to be 0.98. The centroid of mass difference could be amazingly 0.03 and 0.007 cm. In summary, our results demonstrated the feasibility of using this deep learning network for real target tracking in fluoro images, which will be crucial for target verification before or during lung SBRT treatments.