Information Retrieval for AI

From sparse to dense. TF-IDF, BM25, learned sparse retrieval, bi-encoders, late interaction. A spectrum, not a succession.

Where knowledge sits when the system runs. One figure, redrawn every few weeks.

The course opens with attention itself as retrieval over the context, so the whole spectrum reads as one idea at different scales.

The middle three rows are where the figure changed in 2026. Routing computation and routing a lookup used to be one row, until a result that year found them complementary rather than alternative, with an optimum between them. The row above them did not exist at all.

Students are divided into fixed discussion groups of five, held all semester. Each member takes a different role for the week, and roles rotate. Every student holds each role at least once over the term.

You choose your own supporting paper. Nobody assigns it. Recommendations are published for every role on every assigned paper, and taking one costs you nothing - the course pays for what you do with the paper, not for the effort of finding it. The full rules are below.

There are five roles, not six. On the four assigned papers published in the last eighteen months - sessions 15, 21, 25 and 27 - the Replicator becomes the Referee, because no reproduction study exists for a paper that new. The role keeps the same question - should this be believed - and changes the evidence. Where the paper has a public review thread, you read what four reviewers decided and compare it with what your group decided. Where it does not, you write the first review yourself.

Sessions 15 and 27 have threads. Session 21 does not, so the Referee reads the reviews of the contradicting paper as a worked example and then reviews the anchor cold. Session 25 has neither, and that Referee writes a review with no model to follow at all.

Note-taking is a duty rather than a role. Rotate it inside the group.

Everyone reads the assigned paper closely. Each student additionally consults one supporting paper chosen for their role - the abstract, the relevant table, the part bearing on their role. The supporting paper belongs to the role, not the group, so all students holding a role in a given week read the same one. They do not coordinate, and there is no joint deliverable.

One Wednesday therefore puts six papers into circulation: the assigned paper, read by everyone, and five supporting papers.

One review per student, half a page, due 18:00 on Tuesday. That is the entire weekly obligation outside class.

1. Model check. Give a language model the assigned paper and the prompt for your role. Name the model and version, paste its answer in two or three lines, then say what it got wrong or missed.

2. What you found, in your role this week. The substantive point, and what would be needed to believe or refute it. This is the bulk of the page.

3. A class-local hook. One sentence tying the paper to something only a class member knows.

4. One question for the class. Something you cannot answer from the paper. Three are chosen to open Wednesday.

The half-page cap is deliberate. It holds the weekly load down and forces you to pick your best point rather than list five.

The 17 minutes in groups are not open discussion. There is one forced decision and a deadline.

The written conclusion is four lines.

1. The strongest point in the group this week, and which role brought it.
2. What would settle it - the experiment, the number, the missing baseline.
3. Does the paper survive? Yes / yes with caveats / no.
4. Where it sits on the knowledge-location spectrum, in one phrase.

The ranking criterion is announced at 9:00 and varies weekly. Examples: which point most threatens the central claim; which would most change whether you would use this method; which would the authors find hardest to answer; which will still matter in five years; which would you spend a week testing.

During the open challenge, the member who answers is named at the moment the group is called. Every member must therefore be able to defend the group’s conclusion. There are no prepared presentations of the assigned paper. Everyone has read it, and a summary would waste eight minutes of sixty.

Three questions, answered individually in the last two minutes.

1. Which role’s point won in your group?
2. Name one point that was dropped, and why it lost.
3. What did you walk in believing was strongest? Did that change, and what changed it?

Question 2 can only be answered by someone who followed the argument. You read one supporting paper; the other four reach you only through the discussion. Question 3 is the only place the course pays for changing your mind.

To say that system A beats system B, you must know which documents are relevant for a set of queries. That list is a test collection, and a human made it by hand. Every number in every retrieval paper rests on one.

Most students read “nDCG 0.43” as a measurement. It is not. It is the output of a chain of human decisions about what counts as relevant, and you only see that by making those decisions.

The collection is built from the papers the class has read - about 48 documents by December. You cannot judge relevance for documents you do not understand, so a public benchmark in an unfamiliar domain would produce guesses rather than disagreements worth studying.

What a query looks like matters.

At least one conceptual query per student is required.

The numbers work out at roughly 60 queries, 480 query-document pairs, and two judges per pair, which is about 32 judgements each - some 15 minutes of work. Two judges per pair is what makes agreement measurable. Expect agreement to be low, and treat that as the finding rather than the failure.

At 48 documents the metrics are noisy and rankings will flip on reruns. That is also part of the lesson.

Groups claim one modification each at session 9, so that no two duplicate.

• Hybrid BM25 and dense retrieval, fused with reciprocal rank fusion
• A different bi-encoder checkpoint
• doc2query expansion over the collection
• An LLM reranker over the top ten
• Chunking papers into sections instead of indexing whole documents
• ColBERT late interaction

Deliverables. A run file (query_id, doc_id, rank, score), a one-page write-up due a week before session 29, and an eight-minute presentation.

The relevance judgements freeze at session 19. Groups may keep refining afterwards, and should report both their session-19 number and any later one.

Eight reviews, one per discussion session. A random four are marked on the scale below and the rest on completion. You will not know in advance which count.

Note that “did the work adequately” is a 2, not a 4.

Eight sets of three questions, scored against your group’s filed conclusion. Question 1 and question 3 are checked mechanically. Question 2 passes if it names a point actually raised in your group.

Scored 0-2 whenever you are named to speak in plenary, either to defend your group’s conclusion or to answer a question during synthesis. Opportunities are tracked and allocated so that every student is called roughly three times.

Marks go to the response to challenge. Not to fluency, and not to being right.

Scored out of 10, then adjusted by peer evaluation within the group.

Position on the leaderboard is worth 10% of the project mark and 2.5% of the final grade. This is deliberate. A modification that made things worse, correctly diagnosed against a wrong prediction, outscores one that helped by accident.

If your modification does not run, you can still score on prediction and diagnosis - up to 6 of 10 - provided you submit an honest account of what broke. Only the measurement points are lost.

Peer evaluation. Each member distributes 100 points across all five members, including themselves. The mean of what the other four give you becomes a multiplier between 0.85 and 1.15.

The weekly model check is where the policy becomes concrete. Each role’s task needs a capability that is weak in a different way.

Every review contributes one line to a class record of model failures - model, version, paper type, failure observed, how it was caught. It is reviewed at sessions 19 and 29. The interesting question is which failures disappear as models improve during the term, and nobody knows that in advance.

The instructor prepares lecture material with model assistance and adds the model’s mistakes to the same record.

Supporting papers are not assigned. Each week you hold one role and you choose the paper that serves it.

Every role, every week.

1. Declare it. Your review opens with one line before the four fields: Supporting paper: author, year, title, venue, URL or DOI. Why it fits the role: one sentence. No declaration line is marked as if no supporting paper was read.
2. Choose by Tuesday 18:00. No approval step, no earlier deadline. The review is the moment of choice.
3. It must be checkable. A DOI, an arXiv id, or a stable URL.
4. One paper. If you read more, declare the one you rely on.
5. Not the assigned paper, and not another of the eight. Everyone reads those already.
6. No reuse. A paper declared in an earlier week cannot be declared again, in any role.
7. Duplicates between students are fine. No coordination required, no penalty.
8. Papers only, with two exceptions: blog posts, documentation, code and review threads are admissible for Applier and Referee, where the artefact type is the point.
9. English, or translate the passage you rely on.
10. Authorship is not a bar. A later paper by the assigned paper’s own authors is often the right Extender choice.
11. If nothing fits, take the recommendation and say why - one sentence on what you searched for and why it failed. This scores 2. An honest failed search is a result.

What full marks require, by role.

How the choice is marked. It gates the review score rather than carrying its own. A paper that fails its role test caps the review at 1, however good the writing. A paper that passes with the actions missing caps it at 2. Above that the normal scale applies.

A case these rules do not cover is resolved in your favour for that week, and the rules are amended before the next session. Raise it on Slack so the answer reaches everyone.

These are recommendations, not assignments. Each is a safe choice that passes its role’s test, and taking one carries no advantage and no penalty. The rest are published with the roster two weeks before each session.

Everyone holding the Referee role also reads the NeurIPS reviewer guidelines. Two pages, and it is what the role is being asked to do.

Session 9 will produce low agreement between judges, and these explain why that is a result rather than a failure.

• Voorhees (2000), Variations in relevance judgments and the measurement of retrieval effectiveness (IP&M). Required. TREC’s own finding that judges disagree substantially yet system rankings stay stable.
• Zobel (1998), How Reliable Are the Results of Large-Scale IR Experiments? (SIGIR).
• Buckley & Voorhees (2004), Retrieval Evaluation with Incomplete Information (SIGIR).

Reference text. Manning, Raghavan & Schütze, Introduction to Information Retrieval (Cambridge University Press, 2008). Free online. The classical retrieval chapters cover sessions 2, 3, 5 and 7.

Where knowledge sits, the classical lineage. Weston, Chopra & Bordes (2014) Memory Networks; Graves et al. (2014) Neural Turing Machines; Graves et al. (2016) DNC (Nature); Lample et al. (2019) Large Memory Layers with Product Keys; Shazeer et al. (2017) Outrageously Large Neural Networks; Fedus, Zoph & Shazeer (2022) Switch Transformers (JMLR); Jiang et al. (2024) Mixtral of Experts; He (2024) Mixture of a Million Experts.

Where knowledge sits, 2025 and 2026. Berges et al. (2024) Memory Layers at Scale; Cheng et al. (2026) Conditional Memory via Scalable Lookup - the Engram paper, and session 21’s anchor; Behrouz, Zhong & Mirrokni (2025) Titans (NeurIPS); Behrouz et al. (2025) Nested Learning (NeurIPS); Eyuboglu et al. (2025) Cartridges; Lin et al. (2025) Continual Learning via Sparse Memory Finetuning, which is the counter-example to this course’s own argument that weights cannot forget.

Attention as retrieval. Wu et al. (2024) Retrieval Head Mechanistically Explains Long-Context Factuality, the empirical version of an analogy this course used to assert; Zhang & Bottou (2025) Memory Mosaics at scale (NeurIPS).

Long context versus retrieval. Liu et al. (2023) Lost in the Middle (TACL); Li et al. (2024) Retrieval Augmented Generation or Long-Context LLMs? (EMNLP).

Dense retrieval, and one space for two jobs. Reimers & Gurevych (2019) Sentence-BERT (EMNLP); Karpukhin et al. (2020) DPR (EMNLP); Ni et al. (2022) Large Dual Encoders Are Generalizable Retrievers (EMNLP); Muennighoff et al. (2024) Generative Representational Instruction Tuning; Zhang et al. (2024) OneGen (EMNLP Findings); Lee et al. (2025) Gemini Embedding; Zhang et al. (2025) Qwen3 Embedding.

Retrieval that needs reasoning. Su et al. (2025) BRIGHT (ICLR), session 13’s anchor; Shao et al. (2025) ReasonIR.

Multi-vector retrieval, and making it servable. Khattab & Zaharia (2020) ColBERT (SIGIR); Santhanam et al. (2022) ColBERTv2 (NAACL) and PLAID (CIKM); Dhulipala et al. (2024) MUVERA (NeurIPS); Veneroso et al. (2025) CRISP; Faysse et al. (2025) ColPali (ICLR).

Nearest neighbour search and systems. Malkov & Yashunin (2018) HNSW (TPAMI); Jégou, Douze & Schmid (2011) Product Quantization; Johnson, Douze & Jégou (2019) Billion-scale similarity search with GPUs.

Learning to rank and click bias. Burges (2010) From RankNet to LambdaRank to LambdaMART; Joachims et al. (2017) Unbiased Learning-to-Rank with Biased Feedback (WSDM); Craswell et al. (2008) An Experimental Comparison of Click Position-Bias Models (WSDM).

Retrieval-augmented generation. Lewis et al. (2020) RAG; Borgeaud et al. (2022) RETRO; Khandelwal et al. (2020) kNN-LM (ICLR); GraphRAG (2024); Gutiérrez et al. (2025) From RAG to Memory (ICML), which is HippoRAG 2.

Agentic retrieval and deep research. Yao et al. (2023) ReAct; Trivedi et al. (2023) IRCoT (ACL); Asai et al. (2023) Self-RAG; Jin et al. (2025) Search-R1 (COLM), session 25’s anchor; Chen et al. (2025) BrowseComp-Plus, which re-evaluates that literature on a fixed corpus and finds much smaller numbers; Liu, Zhang & Liang (2023) Evaluating Verifiability in Generative Search Engines (EMNLP Findings), on whether citations support the claims attached to them.

Agent memory, and whether it can be measured. Hu, Wang & McAuley (2026) Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR), session 27’s anchor; Wang (2026) MemDelta, on hidden confounds in that evaluation; Anthropic, Effective context engineering for AI agents, read critically as a primary source.

Semantic search in practice. Cormack, Clarke & Buettcher (2009) Reciprocal Rank Fusion (SIGIR); Formal et al. (2021) SPLADE; Nogueira et al., doc2query.

Adversarial retrieval. Greshake et al. (2023) Not what you’ve signed up for (AISec); Zou et al. (2024) PoisonedRAG.

Commercial search. Shah & Bender (2022) Situating Search (CHIIR), the strongest case against replacing search with language models, and worth reading precisely because this course argues the other way; Google’s How Search Works, read critically as a primary source.

Scholarly search and literature synthesis. Lo et al. (2020) S2ORC (ACL); Swanson (1986) on literature-based discovery; Wang et al. (2024) SciMON (ACL); Asai et al. (2024) OpenScholar; Skarlinski et al. (2024) PaperQA2. This block lost its own lecture in the 2026 revision and is now read alongside session 24, because deep research and literature synthesis are the same problem.