BrainTumorNet: Multi-task learning for joint segmentation of high-grade glioma and brain metastases from MR images

Detection and segmentation of primary and secondary brain tumors are crucial in Radiation Oncology. Significant efforts have been dedicated to devising deep learning models for this purpose. However, development of a unified model for the segmentation of multiple types of tumors is nontrivial due to high heterogeneity across different pathologies. In this work, we propose BrainTumorNet, a multi-task learning (MTL) scheme for the joint segmentation of high-grade gliomas (HGG) and brain metastases (METS) from multimodal magnetic resonance imaging (MRI) scans. We augment the state-of-the-art DeepMedic1 architecture using this scheme and evaluate its performance on a highly unbalanced hybrid dataset comprising 259 HGG and 58 METS patient-cases. For the HGG segmentation task, the network produces a Dice score of 86.74% for whole tumor segmentation, which is comparable to 87.35% and 87.19% by the task-specific and single-task joint training baselines, respectively. For the METS segmentation task, BrainTumorNet produces an average Dice score of 62.60% thus outperforming the scores of 19.85%, 57.99%, 59.74%, and 44.17% by the two transfer-learned, task-specific, and single-task joint training baseline models, respectively. The trained network retains knowledge across segmentation tasks by exploiting the underlying correlation between pathologies. At the same time, it is discriminative enough to produce competitive segmentations for each task. The hard parameter sharing in the network reduces the computational overhead compared to training task-specific models for multiple tumor types. To our knowledge, this is the first attempt towards developing a single overarching model for the segmentation of different types of brain tumors.