Latent Space Modelling of a Biomedical Multimodal Dataset

Clinical decision-making is rarely based on a single data source or modality. However, contemporary machine learning applications are mostly tailored to one specific modality. In this work, multimodal data integration, including histologic images and surgery-related texts, is tested for creating patient-level vector representations with information density comparable to that of pathologic analysis for Head and Neck (HNC) patients. Here, we rely on the multimodal HNCfocusing HANCOCK dataset. First, we transform individual modalities into vector representations using proven expert models tailored to their data types. Next, these are aligned into a joint latent space. Finally, the modality-specific vectors are recombined into a single patient-level embedding. With this approach, a mean AUC score of 0.67 for cancer-specific survival and 0.66 for 3 -year recurrence was achieved using only text and image data, compared to 0.79 when including structured pathologic data.