Multimodal prior knowledge determines false memory formation

Memory formation is a complex phenomenon shaped by various experiential traces, yet their exact contributions remain unclear. This study investigates the generation of false memories leveraging different data-driven computational models to independently quantify language-based and vision-based experiential knowledge, as extracted from large-scale databases consisting of 639 billion words and 15 million images, respectively. We then tested the effects of these knowledge sources in two false-memory experiments, one employing images and the other words as stimuli. Our findings unveil both modality-independent and modality-dependent processes in the formation of memory traces. Indeed, we observed a contribution of both prior visual and linguistic knowledge regardless of the types of stimuli to be memorised. However, the extent of this contribution differed as a function of the modality tested: visual prior knowledge is more influential in image-based tasks, while linguistic prior knowledge dominates in word-based tasks. This dual and modality-dependent contribution underscores the adaptive nature of memory processes, revealing the dynamic integration of diverse experiential traces in false memory formation.