Latent Diffusion-Based Visual-Textual Fusion for Survival and Recurrence Prediction of Colorectal Liver Metastases

Colorectal cancer is one of the most prevalent malignancies worldwide, with liver metastases (CLM) critically impacting patient outcomes. Current prediction models for CLM typically rely on whole slide images (WSIs) and clinical data, rather than fully integrating heterogeneous data sources such as pathology reports, which limits their ability to predict patient outcomes accurately. We introduce a novel multimodal framework for the survival and recurrence prediction of CLM that leverages latent diffusion models (LDMs) and large language models (LLMs), combining visual and textual information. Our approach integrates dual-resolution WSIs with detailed pathology reports to predict critical clinical endpoints, namely Time-to-Recurrence (TTR) and overall survival (OS). In the visual branch, a Mean Teacher model processes WSIs to generate precise classification maps that identify key Regions of Interest (ROIs), which are further refined using an advanced LDM to extract robust, highlevel features. Concurrently, pathology reports are analyzed with multiple LLMs to obtain dense semantic embeddings that capture essential clinical parameters, including tumor grade, histopathological patterns, and biomarker profiles. These complementary features are fused using a learned weighting strategy and incorporated into a Cox proportional hazards model for survival prediction. Evaluated on a dataset of 2,118 histological slides from 326 patients, our model achieves a concordance index (C -index) of $0.72 \pm 0.02$ for TTR and $0.78 \pm 0.02$ for OS. Ablation experiments demonstrate the importance of LLM-derived textual features to improve the prognostic performance, improving by 5 % and 6 % in TTR and OS prediction accuracy, respectively, to state-of-the-art methods. These results underscore the potential of our approach to enhance risk stratification and guide personalized treatment planning. Our findings motivate further validation in a large prospective study to confirm the translational impact of this multimodal framework.