DharmaBench: Evaluating Language Models on Buddhist Texts in Sanskrit and Tibetan
DharmaBench: Evaluating Language Models on Buddhist Texts in Sanskrit and Tibetan
Kai Golan Hashiloni,Shay B. Cohen,19 Authors,K. Bar
TLDR
DharmaBench is introduced, a benchmark suite comprising 13 classification and detection tasks grounded in Buddhist textual traditions: six in Sanskrit and seven in Tibetan, with four shared across both.
Abstract
We assess the capabilities of large language models on tasks involving Buddhist texts written in Sanskrit and Classical Tibetan— two typologically distinct, low-resource historical languages. To this end, we introduce DharmaBench , 1 a benchmark suite comprising 13 classification and detection tasks grounded in Buddhist textual traditions: six in Sanskrit and seven in Tibetan, with four shared across both. The tasks are curated from scratch, tailored to the linguistic and cultural characteristics of each language. We evaluate a range of models, from proprietary systems like GPT-4o to smaller, domain-specific open-weight models, analyzing their performance across tasks and languages. All datasets and code are publicly released, under the CC-BY-4 License and the Apache-2.0 License respectively, to support research on historical language processing and the development of culturally inclusive natural-language-processing systems.
