One-shot labeling for automatic relevance estimation
S MacAvaney, L Soldaini - Proceedings of the 46th International ACM …, 2023 - dl.acm.org
Dealing with unjudged documents (" holes") in relevance assessments is a perennial
problem when evaluating search systems with offline experiments. Holes can reduce the …
problem when evaluating search systems with offline experiments. Holes can reduce the …
Evaluating generative ad hoc information retrieval
Recent advances in large language models have enabled the development of viable
generative retrieval systems. Instead of a traditional document ranking, generative retrieval …
generative retrieval systems. Instead of a traditional document ranking, generative retrieval …
Perspectives on large language models for relevance judgment
When asked, large language models~(LLMs) like ChatGPT claim that they can assist with
relevance judgments but it is not clear whether automated judgments can reliably be used in …
relevance judgments but it is not clear whether automated judgments can reliably be used in …
Query performance prediction using relevance judgments generated by large language models
Query performance prediction (QPP) aims to estimate the retrieval quality of a search system
for a query without human relevance judgments. Previous QPP methods typically return a …
for a query without human relevance judgments. Previous QPP methods typically return a …
LLMs Can Patch Up Missing Relevance Judgments in Evaluation
Unjudged documents or holes in information retrieval benchmarks are considered non-
relevant in evaluation, yielding no gains in measuring effectiveness. However, these missing …
relevant in evaluation, yielding no gains in measuring effectiveness. However, these missing …
[PDF][PDF] Team openwebsearch at CLEF 2024: QuantumCLEF
We describe the OpenWebSearch group's participation in the CLEF 2024 QuantumClef IR
Feature Selection track. Our submitted runs focus on the observation that the importance of …
Feature Selection track. Our submitted runs focus on the observation that the importance of …
[PDF][PDF] Team openwebsearch at clef 2024: Longeval
We describe the OpenWebSearch group's participation in the CLEF 2024 LongEval IR track.
Our submitted runs explore how historical data from the past can be transferred into future …
Our submitted runs explore how historical data from the past can be transferred into future …
[PDF][PDF] Open Web Search at LongEval 2023: Reciprocal Rank Fusion on Automatically Generated Query Variants
We describe the participation of the Open Web Search (OWS) team in the shared task
LongEval hosted at CLEF 2023. Our submission is motivated by previous observations on …
LongEval hosted at CLEF 2023. Our submission is motivated by previous observations on …