Course business

Next lecture: Monday 12 October.

Guided paper-search workshop: Wednesday 14 October, 09:00-10:00.

Next week’s workshop awards marks for a tested baseline change

Monday 12 October, 09:00-10:00 KST: search systems lecture. We will cover passage chunking before the workshop.

Wednesday 14 October, 09:00-10:00 KST: guided paper-search modification on Zoom. Everyone makes the same prescribed change to the supplied baseline.

The workshop carries 2 project marks:

• Start with a baseline retriever.
• Predict the effect of a prescribed change.
• Implement the change.
• Submit the run and a short explanation.

Project rules

From hard queries to deep research

Hard queries: How does a passage help solve this problem? Train the retriever or rewrite the query to expose that connection.

Ambiguous queries: Which problem does the user mean? A better ranking cannot supply an unstated intention.

Probabilistic embeddings: Represent uncertainty about possible matches. Then distinguish representation from clarification.

Interaction: Ask for missing intent. Examine Qulac, InteractComp and IntentRL.

Agentic search: Once the task is specified, let an agent choose the next search from the evidence.

Deep research: Scale that loop into an investigation. Evaluate the output, then examine methods and the harness that supports them.

Retrieval with reasoning

Retrieval with reasoning: infer how a document helps solve the query when direct lexical or semantic overlap does not establish its relevance.

Candidate passage: Dissolved salts can accumulate in soil. High salt levels make water uptake harder and can damage roots.

Query for classical IR: “What does soluble salt build-up do to plant roots?”

The passage answers the same question in different words.

Query for retrieval with reasoning: “Can I reuse the water that drains from my plant pots?”

Imagine collecting water in a saucer and pouring it back into the pot. The query does not mention salts. We must connect reuse to salt build-up before we see why the passage helps.

Blue = water. Gold dots = dissolved salts, shown schematically.

The connection: Reuse returns salts to the pot. Repeated reuse can contribute to salt build-up, so the passage explains a possible downside.

BRIGHT, Appendix C, Table 26.

BRIGHT benchmarks retrieval with reasoning

Retrieve a passage that helps solve the problem, even when it uses different words. These are constructed teaching examples, not BRIGHT dataset items.

Example 1. Python: why does 10 come before 2?

Query: “Python sorts my numbers as "1", "10", "2". How do I get "1", "2", "10"?”

Positive passage: sorted(items, key=int) compares each item by its integer value. The returned items keep their original type.

Missing step: These numbers are stored as text. Compare their numerical values to get the intended order.

Hard negative: “Use sorted(items) to put a Python list in ascending order.” This repeats the operation that caused the problem.

Example 2. Maths: 10 people, 3 rooms

Query: “Put 10 people into 3 rooms. What is the smallest possible number in the fullest room?”

Positive passage: The pigeonhole principle: if 10 objects go into 3 boxes, at least one box must contain 4 or more objects.

Missing step: People are objects; rooms are boxes. Three people per room fits only 9. A split of 4, 3, 3 achieves the minimum of 4.

Hard negative: “Count the people in each room to find the fullest room.” This describes an existing arrangement; it does not establish the best possible one.

BRIGHT, §§3.2-3.4.

BRIGHT: more examples of useful passages

Constructed teaching examples, not BRIGHT dataset items.

Example 4. Evidence: is my phone charger broken?

Query: “My phone will not charge from this wall socket. Is the charger broken?”

Positive passage: In this test, the same phone and charger work in another socket. A working lamp also fails in the original socket.

Missing step: Both devices fail at the same socket. The charger works elsewhere. The evidence points to the socket as the problem.

Hard negative: “This charger is compatible with your phone model.” Compatibility does not tell us which part has failed.

BRIGHT, §§3.2-3.4.

These examples identify the desired match: a problem and knowledge that helps solve it. ReasonIR turns that relationship into retriever training data.

ReasonIR trains retrieval for queries that need reasoning

Goal: retrieve knowledge that helps solve a hard query, even when the document explains a principle rather than directly answering the question.

Bi-encoder: encode the query and each document separately, then rank by cosine similarity. Both use the same trainable encoder, adapted from Llama3.1-8B with bidirectional attention.

Training objective: for each query, raise the positive document’s score relative to the hard negative and other documents in the batch. The contrastive loss is the negative log of the positive’s softmax probability over those candidates.

• Query: a problem that needs knowledge.
• Positive: knowledge that helps solve it.
• Hard negative: a plausible topical match that does not help.

What changes? Gradients pass through the query, positive and negative encodings into the shared encoder weights. Both query and document representations change. Cosine similarity is the fixed scoring rule.

Training data: public retrieval examples teach general retrieval; varied-length synthetic examples extend effective query context; hard-query synthetic examples teach the problem-to-knowledge connection. These examples fine-tune the retriever. Llama3.1-70B-Instruct is the separate data generator.

ReasonIR, §§2, 4-5 and Appendix E.1 - official model usage

Trace a hard-query training example

The contrastive objective needs a useful positive and a misleading negative. ReasonIR generates hard-query (HQ) samples that supply that training signal.

1. Select useful knowledge. Start with BRIGHT’s document pool; keep seeds with FineWeb-Edu scores of at least 2. The seed becomes the positive. The filter favours educational content from which a reasoning problem can be built.
2. Generate the problem from the knowledge. Llama3.1-70B-Instruct considers background knowledge, problem-solving patterns and realistic scenarios. It writes a self-contained query without dependence on the seed’s specific terms. This supplies a problem-to-principle link beyond word matching.
3. Generate the distraction separately. Give the LLM the query and positive in another turn. It writes a topical but unhelpful passage. This supplies a contrastive competitor; mining existing passages risks penalising another useful explanation. Separate turns avoid the short, easy negatives observed with joint generation.

Example: circadian-rhythm seed → night-shift sleep query → body-temperature distraction. The loss raises the seed’s similarity relative to the distraction.

BRIGHT’s StackExchange collection: real question → useful passages from answers’ links; Google → topical negatives. An annotator and two domain reviewers must agree. This grounds evaluation in real questions and checked relevance.

Why change the collection process for training? Document-to-query generation supplies new pairs without finding and manually annotating a forum question for each seed. It scales training examples of the desired distinction. Generated relevance can still be wrong; these choices do not guarantee valid labels.

ReasonIR, Figure 3 and §§4.3-4.4 - BRIGHT, §3.2 and Figure 2

Separate the retrieval settings

BM25 is the non-ReasonIR baseline. Compare retrievers with the same query input; then compare rewriting and reranking.

BRIGHT mean nDCG@10 (April 2025):

The reranker is Qwen2.5-32B-Instruct.

ReasonIR, Figure 3 and §§4-5 - Table 1, p. 7.

Holding the query input fixed lets us compare retrievers. Changing the query input can also improve retrieval. Next, RQR trains the model that writes that input.

Write a query that exposes the missing step

Constructed exercise - 3 minutes.

A maze has several routes to the exit. Each move crosses one square and has the same cost. Find a route with the fewest moves.

Write the query you would send to a search engine. Expose the knowledge needed to solve the problem.

You have 45 seconds. We will compare queries before revealing the connection.

RQR learns to reason before retrieval

Goal: Make a hard query more informative for the retriever by spelling out the knowledge it may need.

Approach: Train a language model to reason about the query and write an expanded search query. The retriever then searches with that text.

Original query → reason and rewrite → retrieve documents.

RQR models are called TongSearch QR. It trains the rewriter; ReasonIR trains the retriever. RQR can feed either BM25 or ReasonIR.

How it learns: Reinforcement learning rewards candidate rewrites. Here, the reward measures whether a fixed embedding model scores known useful documents more highly after the rewrite.

Qin et al., EMNLP 2025 - §§3.1-3.2.

Disambiguate the event before answering

AmbigQA’s example: “When do Meredith and Derek get married in Grey’s Anatomy?”

• Informal marriage: season 5.
• Legal marriage: season 7.

Both events provide a defensible answer to a different interpretation. A single answer hides that choice.

Constructed clarification: “Do you mean the informal or legal marriage?” The user replies, “The legal marriage.” Retrieve evidence about that event, then answer season 7.

A document can be accurate and answer the wrong interpretation. Before asking the user, can an embedding at least represent uncertainty about possible matches?

AmbigQA, §§3-4 - Table 1, p. 2.

Probabilistic embeddings represent an input as a distribution

Goal: Represent uncertainty or one-to-many matches in an embedding space.

Approach: Predict a distribution over vectors for each input. Compare distributions to score a possible match.

For this Gaussian example, . Here is an embedding vector; the diagonal covariance uses the predicted coordinate-wise variances.

PCME (Probabilistic Cross-Modal Embedding, CVPR 2021) is one paradigm within probabilistic embeddings. It matches images and text. This is work I co-authored.

The encoder predicts distribution parameters. The next slide shows what a possible image-text match means; the following slide shows how samples produce a match score.

PCME, §3

Constructed image-text example

Text query: “A dog outdoors.”

Image A: A dog runs across a lawn. Image B: A dog rests on a beach. Image C: A cat rests indoors.

Valid matches: A and B depict different scenes that both fit the caption. The caption leaves the activity and setting unspecified.

Encoding: The text encoder returns a mean vector and variances. The image encoder returns a mean vector and variances for each candidate.

Scoring: Sample vectors from the caption and image distributions; aggregate pairwise match scores. Rank the candidate images by that estimate.

What the example illustrates: Several image-text correspondences can be valid. It supplies no trained PCME scores, and the learned spread is not an explicit list of “lawn” and “beach” intentions.

PCME, §3.2.1 and Figure 2

The next slide computes an estimated match probability from sampled vectors. The samples below are supplied for arithmetic; they are not model outputs for these dog images.

Sample pairs estimate a match probability

We now turn the predicted distributions into a score for an image-text pair.

Goal: Combine several possible image-text vector matches into one estimated match probability.

Draw vectors from each distribution. An encoder supplies mean and coordinate-wise variance . A Gaussian draw is

Here is standard Gaussian noise and is coordinate-wise multiplication. Let be image sample and text sample ; and index the samples.

Compare each image sample with each text sample:

The sigmoid maps a distance-based score to a value between 0 and 1. Learn , the distance scale, and , the offset. Average the pair scores to obtain . The distance is Euclidean, not squared Euclidean.

Constructed arithmetic: image draws , text draws , and . These give the distance matrix

With , distance 1 gives ; distance 3 gives .

For a second candidate with image draws , the distances to the same text draws are . Its average pair score is , so this particular sample estimate ranks it above the first candidate.

The draws are supplied for arithmetic. We have not learnt means or variances, run an encoder or measured retrieval accuracy.

PCME, §3 - Eqs. 2-4.

A distribution can represent several valid matches. A match score ranks candidates; it does not reveal which interpretation the user wants. Obtaining that missing context requires interaction.

Ask the user when the gap is their intention

User: “Find papers about memory.”

Retriever: “Do you mean human memory, agent memory, or memory layers?”

User: “Stored history for an LLM agent.”

The answer changes which documents count as relevant. More papers about memory would not, by themselves, reveal this user’s intention.

A clarification answer becomes part of the next retrieval query. Qulac studies that connection between a question, its answer and the resulting ranking.

Qulac: clarification before LLM search agents

SIGIR 2019. Qulac studied whether a system could select a clarification question that improves retrieval for a user’s specific need, or facet.

Original query → select a question → recorded user answer → new document ranking.

The method selects from candidate questions with the query and previous question-answer turns as context. Evaluation uses facet-specific relevance judgements.

Judge clarification by whether the resulting ranking serves the intended need. Next, use a concrete laptop task to distinguish what a system should ask from what it can find in sources.

Aliannejadi et al., Qulac, SIGIR 2019

Ask about intent; search for evidence

Constructed laptop task: “I need a laptop for my PhD. My budget is ₩2 million.”

Ask: “Will you carry it to campus every day? What work will you do on it?”

Student: “I carry it daily and analyse large datasets. My limits are at most 1.5 kg and at least 32 GB RAM.”

The task now has three criteria: budget, weight and RAM.

Search and check: A cheap listing specifies 16 GB RAM. Find the exact 32 GB configuration and its Korean price. This is missing evidence the system can investigate.

A product page can establish weight. It cannot establish whether this student wants to carry that weight every day. The student supplies that intention.

Can an LLM agent make this distinction and ask the useful question? InteractComp tests that capability.

InteractComp, §3.1, ICML 2026 - IntentRL, §3

Clarification exercise!

User: “Find me a laptop for research.”

You are a retriever. You may ask the user one question before you search.

Write the exact question. Name one recommendation decision that its answer could change.

The lecturer will answer one class question. State what you can now search for and what remains unresolved.