Measuring effectiveness of information retrieval (IR) systems is essential for research and
development and for monitoring search quality in dynamic environments. In this study, we
employ new methods for automatic ranking of retrieval systems. In these methods, we merge
the retrieval results of multiple systems using various data fusion algorithms, use the top-
ranked documents in the merged result as the “(pseudo) relevant documents,” and employ
these documents to evaluate and rank the systems. Experiments using Text REtrieval …

Typical information retrieval system evaluation requires expensive manually-collected
relevance judgments of documents, which are used to rank retrieval systems. Due to the
high cost associated with collecting relevance judgments and the ever-growing scale of data
to be searched in practice, ranking of retrieval systems using manual judgments is becoming
less feasible. Methods to automatically rank systems in absence of judgments have been
proposed to tackle this challenge. However, current techniques are still far from reaching the …