Evaluation defines what we want

“Retrieve relevant documents” leaves the need unspecified.

The corpus must fit the need: prose, scientific papers, visual documents or tools.

Takeaway: the user’s need defines relevance; the benchmark chooses how to test it.

Lecturer’s framing; these are illustrative information needs.

Before 2020: define the need before comparing systems

TREC relevance asks whether a document’s information helps with the topic. Lexical overlap is a retrieval signal; the judgement concerns the information need.

Takeaway: a shared test must specify the need, the corpus and the relevance rule.

Selected milestones; later MS MARCO ranking tasks extend the original reading-comprehension collection. | TREC-1, 1992; TREC relevance judgements; CLEF, since 2000; MS MARCO, 2016

TREC-1 and MS MARCO: different needs, different evidence

Takeaway: relevance means that the information performs the job specified by the user’s need.

Topic and passage summaries are paraphrased. The MS MARCO passage states a requirement; it does not establish that the user qualifies. Sources: TREC-1 topic 051; MS MARCO, Figure 1 and §3, PDF p. 4.

2018-2022: test relevance across tasks

BERT (2018) and passage reranking (2019) make pretrained language representations useful for task-specific relevance learning.

Takeaway: relevance learnt for one task must be tested on other tasks and evidence needs.

Selected publication dates; semantic relevance and transfer research predate BERT. MuSiQue is a QA benchmark with supplied contexts. | BERT, 2018; BERT passage reranking, 2019; SciDocs / SPECTER, 2020; BEIR, 2021; MuSiQue, TACL 2022

SciDocs and BEIR: the task changes usefulness

The SciDocs pair crosses application topics through a recorded citation. The label alone does not establish a useful new method transfer.

BEIR combines such tasks to test generalisation. Some needs also require several connected facts; an intermediate fact can be a relevant result.

Takeaway: a transfer test must vary what makes a result useful.

Paraphrased dataset records. SciDocs query 784953… → positive 632589…; FiQA test query 8. | SciDocs / SPECTER, 2020; BEIR, 2021; BEIR SciDocs data; SciDocs relevance labels; BEIR FiQA data

MuSiQue: connect facts to answer the question

Some needs go beyond a useful single result. MuSiQue constructs QA questions that depend on connected facts in supplied passages.

Question: Who did the person portraying Batman in the Justice League movie play in Dazed and Confused?

Answer: Fred O’Bannion.

Takeaway: an intermediate fact is useful because it connects the question to the answer.

MuSiQue, TACL 2022. MuSiQue-Ans dev, 2hop__72118_83769. Original question; supporting passages paraphrased.

2022-2024: what must retrieval give an LLM?

An LLM needs retrieved material that supports the user’s task. These examples make four requirements concrete.

Examples: FollowIR; BRIGHT; AnaloBench; MultiHop-RAG.

FollowIR: satisfy the instruction

Query: find positive achievements of the Hubble telescope.

Stricter condition: credit achievements that relied solely on Hubble.

Takeaway: a topical match must also obey the relevance instruction.

Figure 2; paraphrased paper illustration. | FollowIR, NAACL 2025

BRIGHT: find explanatory knowledge

BRIGHT and AnaloBench both require an underlying connection. This BRIGHT example connects an observation to a biological mechanism.

Query (excerpt): “How does a tree trunk sprout and grow after being cut?”

Takeaway: to judge relevance, identify the mechanism behind the observation.

Table 20, PDF p. 32; original query excerpt and paraphrases of the labelled positive and negative documents. | BRIGHT, 2024 / ICLR 2025

AnaloBench: retrieve useful analogies

Task: given a story, retrieve analogous stories from a story bank.

Shared relation: a provider loses the ability to support others.

Takeaway: retrieve the shared relationship even when the characters and causes differ.

Figure 1, PDF p. 1; the paper’s Maria/oak example, with both stories paraphrased. | AnaloBench, EMNLP 2024

Related: research inspiration

MultiHop-RAG: assemble sufficient evidence

Query, paraphrased: does Fortune explain rate rises through past conditions, while the Sydney Morning Herald describes decisions based on incoming data?

The paper’s answer is yes. One article alone leaves half the comparison unsupported.

Takeaway: judge the retrieved set by whether it supports the complete answer.

Table 1, PDF p. 2; paraphrased historical news example. | MultiHop-RAG, 2024

Related: connected facts in MuSiQue

Multilinguality: preserve the need and its conditions

MMTEB broadens evaluation across languages; mFollowIR tests instruction following across languages. CRUMB tests combined requirements.

Takeaway: preserve both the requested fact and its conditions across languages.

Gemini Figure 2; reported passages, not a current state-bird claim. CRUMB Figure 1 and Legal QA example; the wrong-jurisdiction candidate is a classroom contrast. | MMTEB, 2025; Gemini Embedding, 2025; mFollowIR, 2025; CRUMB, 2025

Context: recover implicit meaning

Useful evidence may require context that the query or isolated chunk leaves unstated.

BRIGHT linked an observation to a mechanism. Here the link depends on a remembered pattern or surrounding text.

Takeaway: relevance can depend on implicit relations and context beyond the retrieved sentence.

OBLIQ Figure 2; ConTEB Figure 2 and introduction, PDF pp. 1-2. Paraphrased illustrations except the short ConTEB sentence. | OBLIQ-Bench, 2026; ConTEB, 2025; BRIGHT, 2024 / ICLR 2025

Tool retrieval: find a capability for the next action

An agent may need evidence for an answer or a capability for an action.

ToolRet sample: cancel a smart-device timer, with device ID and switch-time inputs and an operation status.

A timer-creation tool shares the topic but does not cancel.

Takeaway: tool relevance depends on the required operation; successful execution is a separate outcome.

ToolRet PDF p. 22, seed-instruction illustration; the timer-creation candidate is a classroom contrast. | BrowseComp-Plus, 2025; ToolRet, 2025

Evidence search: resolve the clues

For evidence search, the retrieved set must resolve the clues that identify the answer.

Question, shortened: a heritage society found a lost bridge near its village. The village’s name comes from a river and a walking stick. What is its by-name?

Takeaway: judge support for the answer-determining clues; a detail such as the bridge’s age can be confirmatory.

Figure 1, PDF p. 2, QID 234. Shortened question and paraphrased evidence from the BrowseComp-Plus projection onto ClimbMix. | ClimbMix projection, Aug 2026

Research inspiration: find a method that serves the goal

Retrieval can help develop an idea as well as answer a factual question. AnaloBench’s shared-relation requirement becomes a search for useful research methods.

MIR proposal: reduce BERT’s size while preserving its performance.

ResearchBench also retrieves scientific inspirations for hypothesis development.

Takeaway: a useful research connection must serve the proposed goal.

MIR Figure 1; ResearchBench task definition. The method contrast is a paraphrased paper illustration. | MIR, ACL 2025; ResearchBench, ACL Findings 2026; AnaloBench, EMNLP 2024

Research inspiration: address, broaden or specify the problem

A useful research connection also depends on the kind of help requested.

Problem: an RL-trained model exploits its reward signal.

Takeaway: judge whether the connection performs the requested relation to the problem.

Introduction; motivating illustrations, not verified test records. | RATIO, Aug 2026

Creative connections: reward specificity and diversity

Useful connections can have several valid routes. CREATE asks how Dakota Johnson connects to people who starred in fantasy or sci-fi films.

CREATE generates paths from model knowledge. An evidence-retrieval extension would need sources for the links.

Takeaway: creative utility rewards specific, valid connections and diversity across paths.

Figure 1, PDF p. 2; paraphrased motivating question and paths. | CREATE, 2026

Freshness: recheck evidence as sources change

Useful evidence must also remain valid when the corpus changes.

Still Fresh? tests FreshStack questions against documentation snapshots from 2024 and 2025. Some supporting material moves from LangChain to repositories such as LlamaIndex.

Takeaway: renew evidence judgements as sources change; a new URL alone does not imply a new relevance decision.

Abstract and temporal analysis. | Still Fresh?, 2026

From needs to scores: define relevance first

The benchmark examples define different needs. To compare rankings, fix one need and its relevance rule.

Constructed example: explain why document padding can lower a BM25 score.

For binary evaluation, grades 2 and 3 count: relevant set {a, b, c}.

Precision@k = relevant results in top k / k

Recall@k = relevant results in top k / total relevant documents

Takeaway: the relevance rule determines which results receive credit.

Classroom grades are stipulated; they are not benchmark measurements.

R-Precision: let the relevant set choose the cutoff

Let R be the number of relevant documents for this query, according to the relevance judgements.

R-Precision = Precision@R
= relevant documents in the top R / R

Our relevant set is {a, b, c}, so R = 3.

At k = R, precision equals recall. The cutoff can differ between queries.

For R > 0. When averaging here, omit queries with R = 0 and report the omitted count. Introduction to Information Retrieval, §8.4. Constructed example.

MRR: how early is the first relevant result?

Reciprocal rank (RR) = 1 / rank of the first relevant result.

Mean reciprocal rank (MRR) = the unweighted mean of RR across queries.

With a cutoff k, use RR@k = 0 if the top k results contain no relevant document.

Our relevant set is {a, b, c}.

Both score 1 for this query. Later relevant results do not increase RR.

Specify the cutoff when reporting MRR@k. Reciprocal rank definition. Constructed rankings.

Success@k: did the top k contain a relevant result?

For one query, Success@k is 1 if the top k contains at least one relevant document; otherwise it is 0.

Across queries, its mean is the fraction of queries with a hit.

Our relevant set is {a, b, c}. Use k = 3.

Success asks whether anything relevant was found. Recall asks what fraction of the relevant set was found.

Success@k definition. Constructed rankings.

DCG assigns value to grade and rank

To reward a better order, give each relevance grade a gain and discount later ranks.

DCG@k = Σ_i=1…k g(r_i) / log₂(i+1)

Here r_i is the grade at rank i. Discount weights begin 1, 0.6309, 0.5.

Takeaway: the gain and discount encode what a good ranking should reward.

The gain and discount used throughout the worked example.

nDCG compares a ranking with its ideal

The ideal DCG depends on the available relevance grades. Normalise each query against its own ideal.

nDCG@k = DCG@k / IDCG@k

IDCG uses the best order of the same grades, with the same gain, discount and cutoff.

Across queries: use the unweighted mean. Here, omit queries with IDCG=0 and report the omitted count.

Takeaway: report the gain, cutoff, aggregation and zero-gain convention with the score.

Constructed example. Divide using unrounded values. nDCG definition.

Efficiency: how much does retrieval quality cost?

Measure retrieval quality against five separate cost axes.

It’s difficult to compare costs across different categories above, as the frequency of computation instances differs across categories. Query-specific ones are rather recurrent, while index and infrastructure costs are incurred only sporadically. We recommend reporting these numbers seperately.

Example efficiency benchmark: Big-ANN