Towards explainable evaluation metrics for machine translation
Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics for
machine translation (for example, COMET or BERTScore) are based on black-box large …
machine translation (for example, COMET or BERTScore) are based on black-box large …
IndicMT eval: A dataset to meta-evaluate machine translation metrics for Indian languages
The rapid growth of machine translation (MT) systems necessitates meta-evaluations of
evaluation metrics to enable selection of those that best reflect MT quality. Unfortunately …
evaluation metrics to enable selection of those that best reflect MT quality. Unfortunately …
Extrinsic evaluation of machine translation metrics
Automatic machine translation (MT) metrics are widely used to distinguish the translation
qualities of machine translation systems across relatively large test sets (system-level …
qualities of machine translation systems across relatively large test sets (system-level …
Reranking for natural language generation from logical forms: A study based on large language models
Large language models (LLMs) have demonstrated impressive capabilities in natural
language generation. However, their output quality can be inconsistent, posing challenges …
language generation. However, their output quality can be inconsistent, posing challenges …
Navigating the metrics maze: Reconciling score magnitudes and accuracies
Ten years ago a single metric, BLEU, governed progress in machine translation research.
For better or worse, there is no such consensus today, and consequently it is difficult for …
For better or worse, there is no such consensus today, and consequently it is difficult for …
Metric score landscape challenge (MSLC23): Understanding metrics' performance on a wider landscape of translation quality
Abstract The Metric Score Landscape Challenge (MSLC23) dataset aims to gain insight into
metric scores on a broader/wider landscape of machine translation (MT) quality. It provides a …
metric scores on a broader/wider landscape of machine translation (MT) quality. It provides a …
ACES: Translation accuracy challenge sets at WMT 2023
We benchmark the performance of segmentlevel metrics submitted to WMT 2023 using the
ACES Challenge Set (Amrhein et al., 2022). The challenge set consists of 36K examples …
ACES Challenge Set (Amrhein et al., 2022). The challenge set consists of 36K examples …
MT-Ranker: Reference-free machine translation evaluation by inter-system ranking
Traditionally, Machine Translation (MT) Evaluation has been treated as a regression
problem--producing an absolute translation-quality score. This approach has two limitations …
problem--producing an absolute translation-quality score. This approach has two limitations …
Towards fine-grained information: Identifying the type and location of translation errors
Fine-grained information on translation errors is helpful for the translation evaluation
community. Existing approaches can not synchronously consider error position and type …
community. Existing approaches can not synchronously consider error position and type …
BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation
Although neural-based machine translation evaluation metrics, such as COMET or BLEURT,
have achieved strong correlations with human judgements, they are sometimes unreliable in …
have achieved strong correlations with human judgements, they are sometimes unreliable in …