Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged - Apple Machine Learning Research

research area

Speech and Natural Language Processing

conference

EMNLP

content type

paper

published

October 2026

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Authors Hala Almaghout, Christian Federmann, Qin Gao

View publication

Copy Bibtex

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

Related readings and updates.

Understanding Annotator Safety Policy with Interpretability

July 6, 2026 research area Fairness , research area Methods and Algorithms conference ACM Conference on Fairness, Accountability, and Transparency

Safety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures (annotators misunderstand or misexecute the task), policy ambiguity (policy wording leaves room for interpretation), or value pluralism (different annotators hold different perspectives on safety). Distinguishing these…

Read more

Generalizable Error Modeling for Human Data Annotation: Evidence from an Industry-Scale Search Data Annotation Program

September 30, 2024 research area Data Science and Annotation conference Journal of Data and Information Quality

Machine learning (ML) and artificial intelligence (AI) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and…

Discover opportunities in Machine Learning.

Our research in machine learning breaks new ground every day.

Work with us