Evaluating Large Language Models: A Review of Metrics and Benchmarks

Recent advances in large language models (LLMs) have enabled natural language processing (NLP) to achieve notable progress in almost all tasks, such as text classification, language generation, and machine translation. However, the evaluation of these models has become a challenge as they grow and become more complex. Traditional metrics for evaluation, such as accuracy, BLEU scores, and perplexity, struggle to capture detailed aspects such as semantic understanding, coherence, and contextual relevance. In this work, we review various approaches to the evaluation of LLMs and provide a well-structured classification made up of four core categories: classification metrics, generation metrics, translation metrics, semantic and contextual metrics. We present a taxonomy of these evaluation systems, outlining their advantages and disadvantages, followed by a practical guide to choosing the metrics most appropriate for specific task needs. Additionally, we provide benchmark suites such as GLUE and SuperGLUE that offer a standardized framework through which the models’ performance can be evaluated across several tasks in NLP. The objective of this study is to offer a clear taxonomy as a guide for researchers and practitioners in selecting optimal metrics for their applications and to outline future directions for enhancing the evaluation methodologies.