Research

Project 1. Multi-Omics Deep Learning Improves FDG PET-CT-based Long-term Prognostication of Breast Cancer

Breast cancer is the most common cancer among women worldwide and a leading cause of cancer-related deaths. FDG PET-CT, which combines anatomical and metabolic information, has significant potential in breast cancer staging and prognosis. However, current approaches rely heavily on manual annotations and fail to fully integrate multi-modal data for comprehensive survival prediction. In this study, we aim to develop a multi-omics prognostic stratification model (MOPS) that integrates clinical data, medical reports, and PET-CT imaging. By leveraging advanced multi-modal techniques, including cross-modal attention mechanisms and transformer-based language models, our goal is to provide interpretable results, enable personalized risk stratification, and guide tailored treatment strategies, effectively bridging the gap between imaging data and clinical practice.

The model incorporates a 3D ResNet with two branches to process CT and PET independently and utilizes a segmentation model based on STU-Net to generate attention maps that highlight breast cancer tumors and lymph nodes. The 3D ResNet functions as the image encoder, while a text encoder, adapted from RadioLOGIC, encodes medical reports into 768-dimensional vectors. Clinical data are encoded using a multi-layer perceptron (MLP). Features from all modalities are projected into a 128-dimensional space and fused through two cross-modal attention modules: one integrates image and report data into radiomic features, and the other combines radiomic and clinical data. The final risk score is generated through a fully connected layer.

The deep learning-based risk score integrates imaging, medical reports, and clinical information, serving as a new predictor. By processing a large amount of patient diagnostic data, MOPS can capture complex patterns and features that may be overlooked by traditional staging systems. The validation of our model in in-cohort demonstrated its robust prognostic value and significant potential for clinical applications. The stratification of patients into low-, and high-risk groups based on risk score providing more precise predictions of patient outcomes. To further provide support for more refined disease management and treatment strategies, we conducted a comprehensive subgroup analysis. Our model demonstrated robust stratification ability across all molecular subtypes of breast cancer.