Multimodal Question Answering for Botanical Archives: Adapting PaLI-GEMMA 2 to Enhance Exploration of Herbarium Collections

Herbarium collections serve as irreplaceable archives for taxonomic, ecological, and evolutionary research. Recent advances in deep learning now make it possible to design AI models capable of automatically annotating herbarium specimens (Ariouat et al. 2025, Sklab et al. 2025a, Sklab et al. 2025b), to develop intelligent systems for querying, visualizing, and exploring them. In previous work Sklab et al. 2025a, state-of-the-art Computer Vision (CV) models failed to generalize to unseen specimens, relying on superficial cues like labels and mounting paper rather than learning true plant morphology. Meanwhile, rich botanical texts remain disconnected from specimen images: CV models lack semantic depth, and LLMs extract traits but can’t ground them visually.

Vision–Language Models (VLMs)

emerge as a natural solution, combining the strengths of both to close this gap

.

By jointly reasoning across images and text, they have the potential to align visual perception with botanical semantics. However, most existing VLMs are trained on general-purpose datasets and fail to capture the specialized features inherent to herbarium specimens. To address this limitation, we investigate how adapting a VLM to the botanical domain enables it to answer expert-level questions about individual herbarium specimens.

We adapted the

PaliGemma-2

model (Andreas 2024), leveraging its Visual Question Answering (VQA) capabilities to align large language model knowledge with the observable content of herbarium images. In this setup, the digitized specimen is provided as visual context, and the user interacts with the model through targeted questions. The adapted model grounds its answers in visible features of the specimen—morphology, phenology, and context—rather than relying on generic species-level knowledge. By doing so, the system aims to exploit the semantic richness encoded in modern large language models while constraining it to the biological reality captured in a single herbarium sheet — a snapshot of the plant at the moment it was collected, rather than an idealized description of its entire life history. Our approach comprises three stages (Fig. 1): i) domain-specific dataset construction; ii) parameter-efficient fine-tuning; and iii) evaluation.

We curated a dataset from herbarium specimens, combining digitized images with expert-annotations (28 traits). To translate structured trait annotations into a linguistically and semantically rich training corpus, we employed the

LLaMA 4

(Meta AI 2025) language model to automatically generate over 50,000 question–answer pairs grounded in botanical semantics. To achieve domain adaptation under limited computational resources, we employed

Quantization-aware Low-Rank Adaptation (QLoRA)

, which enables efficient fine-tuning by introducing trainable low-rank adapter layers into a quantized pre-trained backbone. We evaluated the fine-tuned model against two baselines, its un-fine-tuned version and the larger

PaliGemma-28B

model, using five metrics (Fig. 2) to quantify both semantic similarity and lexical fidelity with respect to expert annotations. Fine-tuning yielded substantial improvements across all evaluation metrics. The adapted PaliGemma-3B achieved a 40% increase in SBERT similarity (Fig. 2) relative to its base counterpart, reflecting enhanced semantic understanding of botanical traits. Likewise, gains in ROUGE-L and METEOR demonstrate the model’s capacity to employ precise botanical terminology rather than generic descriptive language. This highlights the model’s potential to ground linguistic outputs in visual evidence, enabling more accurate interpretation of herbarium specimens.