Constraints to Neural Machine Translation Quality, Human and Automated Evaluation, and Quality Improvement across Language Pairs: A Systematic Literature Review

There is no systematic literature review (SLR) that has attempted to synthesize current knowledge on Neural Machine Translation (NMT) quality. The objectives of this SLR are to investigate constraints to NMT quality; examine strengths, limitations, and performance of automated and human evaluation metrics; and identify approaches that can be used to improve NMT quality. The PRISMA and SALSA methodologies were adopted to carry out this SLR. Peer-reviewed articles published in English between 2018 and 2024 were searched on the Saudi Digital Library, Web of Science, and Scopus. Furthermore, references of included articles were searched. There were 51 articles spanning 89 language pairs that met the inclusion criteria and were included in this SLR. The major constraints to NMT quality are the morphological diversity of language pairs and low corpora quality and quantity, which are challenges specific to low-resource languages and domains. BLEU is the dominant automated metric, and it is highest in high-resource morphologically diverse languages such as English and Arabic. Moderate BLEU scores were observed in high resource morphologically similar pairs such as European languages and some Asian languages such as Chinese, Japanese, and Korean. Innovative approaches aimed at bridging corpora and morphological diversity have been proposed. Therefore, significant progress has been made in bridging human and NMT performance at the lexical dimension. However, human evaluation showed NMT performance was unsatisfactory in other dimensions, such as adequacy and fluency. NMT has not yet matched human translation, and the focus needs to shift to other language dimensions such as adequacy, fluency, politeness, and context awareness.