This article presents the most up-to-date, influential automated, semiautomated and human
metrics used to evaluate the quality of machine translation (MT) output and provides the …

In this paper, we introduce the Eval4NLP-2021shared task on explainable quality
estimation. Given a source-translation pair, this shared task requires not only to provide a …

Large language models (LLMs) are competitive with the state of the art on a wide range of
sentence-level translation datasets. However, their ability to translate paragraphs and …

This chapter presents an overview of different approaches and tasks related to classification
and analysis of errors in machine translation (MT) output. Manual error classification is a …

We compare three approaches to statistical machine translation (pure phrase-based,
factored phrase-based and neural) by performing a fine-grained manual evaluation via error …

Many NLP researchers are experiencing an existential crisis triggered by the astonishing
success of ChatGPT and other systems based on large language models (LLMs). After such …

In this paper, we first explore the role of inter-annotator agreement statistics in grammatical
error correction and conclude that they are less informative in fields where there may be …

This study comprehensively evaluates the translation quality of Large Language Models
(LLMs), specifically GPT-4, against human translators of varying expertise levels across …

This paper presents a quantitative fine-grained manual evaluation approach to comparing
the performance of different machine translation (MT) systems. We build upon the well …

Inter-Annotator Agreement (IAA) is used as a means of assessing the quality of NLG
evaluation data, in particular, its reliability. According to existing scales of IAA interpretation …