Butterfly-Net: Spatial-Temporal Architecture For Medical Image Segmentation

Radiation therapy tries to maximize the effect of radiation on the tumor and minimize its influence on adjacent tissues. However, it highly depends on the accuracy of the tumor segmentation on the planning radiography images. Tumor contouring today is carried out exclusively with the significant contribution of medical specialists which is a tedious, time-demanding, and expensive task. Further, it is prone to the inter-/intra-observer variation that can affect the reliability of the outcome. Existing methods for automatic tumor segmentation can reduce the influence of these factors, but are not completely reliable and leave a lot of room for improvement. In this work, we exploit Spatio-temporal information from the longitudinal CT scans to improve the deep neural network for tumor segmentation. For this purpose, we devise a novel volumetric Spatio-temporal memory network, Butterfly-Net, which stores the previous scan information and reads for the segmentation at the target time point. Moreover, the effect of clinical factors is investigated in the framework of our volumetric Spatio-temporal memory network. Experimental results on our longitudinal CT scans show that our model could effectively utilize temporal information and clinical factors for tumor segmentation. The code is made publicly available 1