Respiratory motion prediction based on 4D-CT/CBCT using deep learning

Purpose: The purpose is to investigate the feasibility of using Convolutional Neural Network (CNN) to register phase-to-phase deformation vector field (DVF) of lung 4D Computed Tomography (CT) / Cone-Beam Computed Tomography (CBCT).

Methods: A Convolutional Neural Network (CNN) based deep learning method was built to directly register the deformation vector field from the individual phases images of patient 4D CT or 4D CBCT for 4D contouring, dose accumulation or target verification. . The input consists of image pairs while the output is the corresponding DVF that registers the image pairs. The network consisted of four convolutional layers, two average pooling layers and two fully connected layers. The loss function was half mean squared error. The centers of patch pairs are uniformly chosen across the lung and the number of samples were chosen to cover the majority movement of deformable vectors. The method was evaluated by using4-dimensional image volumes from 9 patients with lung cancer, such as 4D-CT, simulated 4D-CBCT reconstructed from DRR, and real 4D-CBCT reconstructed from real projections. In intra-patient study, These image volumes were sortedthe image volumes were sorted into different combinations, (1) training and testing samples from the same 4D-CT image volume, (2) training and testing samples from two 4D-CT volumes, (3) training and testing samples from 4D-CBCT volumes simulated by DRR from 4D-CT volumes, (4) training from 4D-CT and testing from 4D-CBCT reconstructed from primary projections, and (5) training and testing samples from two 4D-CBCT volumes reconstructed from primary projections. In inter-patient study, five 4D-CT volumes from five patients were used as the training set and the sixth patient 4D-CT volume was the testing set. The functionality of a well-trained network adapting new patient’s anatomy was tested. The coefficient of correlations between the prediction DVF and the ground truth DVF and the cross correlations between the target image and the ground truth deformed image, and the target image and the predicted deformed image were calculated. The registered images from predicted DVF and ground truth DVF were reconstructed and compared. One set being the training set and the other the testing set. The centers of patches (i.e. control points) are uniformly chosen across the lung. The limit of physical memory compromises the number of samples and size of patches.

Result: The ratio of cross correlation between predicted deformed image and target image to cross correlation between ground truth image and target image is 0.78 by averaging the intra-patient studies. For inter-patient study, this number is 0.62. In comparing the predicted deformed image and the ground truth image, major features such as diaphragms, lumen and main vessels are matched with each other.

Conclusion: CNN based regression model successfully learn the DVF from one image set, and the trained model can be successfully transferred to another data set, provided the high image quality in training sets and similar anatomic structure between both image sets.