A Comparative Analysis of BLEU and BERT Score for Evaluating Machine Translation of a Telugu Poem

The proliferation of Machine Translation (MT) systems has necessitated robust automatic evaluation metrics. While BLEU (Bi-Lingual Evaluation Understudy) and BERT

(Bidirectional Encoder Representations from Transformers) These are the longstanding standards for assessing lexical and n-gram overlap. Its efficacy in evaluating semantically complex and culturally nuanced texts, such as poetry and prose, remains limited. This study presents a comparative evaluation of five NMT systems—Google Translate, Microsoft Translator, Quill Bot, Gork, and Bhashini on a corpus of Telugu literary texts, the poem "Bandipotlu". This research employs both the traditional BLEU metric and the context-aware BERT metric scores to analyse machine translation quality. Our findings indicate a significant divergence in the rankings provided by these metrics. BLEU scores are heavily penalizing creative paraphrasing and stylistic variations, whereas BERTScores leverage semantic embeddings, and they demonstrate a higher correlation with human intuitions for literary translation. The results suggest that BERTScore is a more suitable metric for evaluating the preservation of meaning, tone, and cultural nuance in literary machine translation, and it advocates for a paradigm shift beyond surface-level n-gram matching.