Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension
Alongside huge volumes of research on deep learning models in NLP in the recent years,
there has been much work on benchmark datasets needed to track modeling progress …
there has been much work on benchmark datasets needed to track modeling progress …
Benchmarking foundation models with language-model-as-an-examiner
Numerous benchmarks have been established to assess the performance of foundation
models on open-ended question answering, which serves as a comprehensive test of a …
models on open-ended question answering, which serves as a comprehensive test of a …
Evaluating open-domain question answering in the era of large language models
Lexical matching remains the de facto evaluation method for open-domain question
answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate …
answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate …
QAFactEval: Improved QA-based factual consistency evaluation for summarization
Factual consistency is an essential quality of text summarization models in practical settings.
Existing work in evaluating this dimension can be broadly categorized into two lines of …
Existing work in evaluating this dimension can be broadly categorized into two lines of …
Crossfit: A few-shot learning challenge for cross-task generalization in nlp
Humans can learn a new language task efficiently with only few examples, by leveraging
their knowledge obtained when learning prior tasks. In this paper, we explore whether and …
their knowledge obtained when learning prior tasks. In this paper, we explore whether and …
A critical evaluation of evaluations for long-form question answering
Long-form question answering (LFQA) enables answering a wide range of questions, but its
flexibility poses enormous challenges for evaluation. We perform the first targeted study of …
flexibility poses enormous challenges for evaluation. We perform the first targeted study of …
Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods
Natural Language Generation (NLG) has made great progress in recent years due to the
development of deep learning techniques such as pre-trained language models. This …
development of deep learning techniques such as pre-trained language models. This …
Neural natural language generation: A survey on multilinguality, multimodality, controllability and learning
Developing artificial learning systems that can understand and generate natural language
has been one of the long-standing goals of artificial intelligence. Recent decades have …
has been one of the long-standing goals of artificial intelligence. Recent decades have …
Towards question-answering as an automatic metric for evaluating the content quality of a summary
A desirable property of a reference-based evaluation metric that measures the content
quality of a summary is that it should estimate how much information that summary has in …
quality of a summary is that it should estimate how much information that summary has in …
Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation
The predictions of question answering (QA) systems are typically evaluated against
manually annotated finite sets of one or more answers. This leads to a coverage limitation …
manually annotated finite sets of one or more answers. This leads to a coverage limitation …