The recording
You were last reading slide 1.
Information Retrieval for AI
How AI systems reach the knowledge they do not hold
Session 1 - Why retrieval now
Mon 31 Aug 2026 - KAIST AI - Seong Joon Oh
The claim
Model weights are a lossy, frozen compression of the world.
Everything a system knows at inference either sits in those weights, or it is fetched.
The fetching path is the subject of this course.
Arguments for retrieval now
Not “retrieval is useful”. Rather: the pressure towards retrieval is increasing, in 2026, for reasons that are structural.
1. Two paths for knowledge
Knowledge takes one of two paths, and they are not equally available:
| Parametric - into weights | Non-parametric - into an index | |
|---|---|---|
| Needs | GPUs and a training run | inference only |
| Needs to know | post-training | how to call an encoder |
| To change one fact | rebuild | write one row |
2. Weights freeze. The world does not.
A checkpoint has a cutoff. The cutoff never moves.
| Fix | Cost | Latency to a new fact |
|---|---|---|
| Retrain | enormous | months |
| Finetune | large | days, and it forgets |
| Index a document | seconds | seconds |
The gap is not 2× or 10×. It is categorical.
3. Cost per long-tail fact
Term frequency is Zipfian. The head is cheap. The tail matters.
- Storing a rare fact in weights costs training compute across all the parameters
- Storing it in an index costs a few hundred bytes
- The tail is where retrieval quality is actually decided
4. Attribution
A parametric answer has no provenance. There is no pointer back.
A retrieved answer carries a document id.
That buys three things weights cannot:
verification - liability - the ability to say “I don’t know, and here is why”
5. Deletion and editing
| Requirement | Weights | Index |
|---|---|---|
| Remove this person’s data | ? | DELETE |
| Per-user knowledge | one model each | one row each |
| Correct one wrong fact | retrain, or finetune and hope | UPDATE one row |
Objections and counterarguments
“Long context will kill retrieval.” Million-token windows exist. Attention still degrades in the middle, cost is still superlinear in practice, and someone still has to choose what goes in the window. That chooser is a retriever.
“Bigger models hold more knowledge.” They do. Scale did not close the out-of-domain gap when BEIR measured it (2021), and the gap reopens once the query needs reasoning (BRIGHT, 2025).
Live: a model reads a paper
I am going to ask a frontier model a question with a checkable answer - not opinion, not textbook lore, a number the paper itself either does or does not report.
The question:
“What does BEIR report as BM25’s average nDCG@10 across the benchmark?”
Watch what comes back, once with nothing attached and once with the paper.
What does BEIR report as BM25's average nDCG@10 across the benchmark?
Asked without the paper
BM25 averages 0.440 nDCG@10 over the 18 datasets. The strongest system, BM25 with a cross-encoder reranker, reaches 0.476. Among standalone dense retrievers TAS-B is around 0.415 and ANCE 0.405. These averages are in Table 4, the zero-shot results table.
Asked with the paper attached
The paper reports no average nDCG@10. Table 2 gives per-dataset nDCG@10 for ten systems across 18 datasets, and closes with a single summary row, "Avg. Performance vs. BM25", expressed as a relative percentage rather than a score: BM25 with a cross-encoder is +11%, docT5query +1.6%, TAS-B -2.8%, ANCE -7.4%.
What the paper actually says
No average nDCG@10 appears anywhere in the paper. 0.440, 0.476 and 0.405 appear nowhere in it, and 0.415 appears only as ANCE on ArguAna, which is not an average of anything. The results table is Table 2, not Table 4; Table 4 is the Hole@10 analysis of TREC-COVID. The dataset table is Table 1, not Table 2.
Try it yourself
Two boxes, one paper. Ask each one anything about BEIR - one answers cold, one reads the paper first.
Without retrieval
With BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
What it got wrong
Nothing on the page told you which one was invented. Both named numbers. Both cited a table. Both were fluent, and neither hedged.
One of them was invented, and the writing did not carry the difference.
Where does the knowledge sit when the system runs?
| Regime | Mechanism |
|---|---|
| 1. Parametric | all in weights, fixed compute |
| 2. Cond. parametric, by compute | a router selects which parameters to run |
| 3. Cond. parametric, by lookup | a hash selects which parameters to read |
| 4. Written at inference | memory learned while the model runs |
| 5. Semi-parametric | an index wired into the forward pass |
| 6. Non-parametric | a corpus placed into the prompt |
| 7. Open world | a policy decides what to fetch |
1. Parametric
All knowledge in weights. Fixed compute per token.
A dense LLM.
- Fastest path: no lookup, no index, no network
- Knowledge is smeared, so nothing is attributable and nothing is deletable
- The cutoff is baked in
Every argument in the first half of this lecture is an argument about the cost of staying here.
2. Conditionally parametric, by compute
A router selects which parameters to run.
Mixture of experts. shazeer-2017-outrageously-large-neural-networks?
Parameters ↑ while FLOPs per token stay flat.
Still fully parametric - all knowledge is in weights. What changed is only which weights are switched on.
3. Conditionally parametric, by lookup
A hash selects which parameters to read.
Product-key memory, memory layers at scale, Engram.
Not “which expert computes” but “which row of a huge table do I fetch”.
2026: routing compute and routing lookup are complementary, not alternative - there is an optimum between them.
4. Written at inference
The memory is learned while the model runs
Titans - cartridges - sparse memory finetuning
This row did not exist when the figure was first drawn.
5. Semi-parametric
An index wired into the forward pass.
kNN-LM, RETRO, memory networks.
The retrieved neighbours are consumed by the model, not by the prompt.
6. Non-parametric
A corpus placed into the prompt
Retrieval-augmented generation
No gradient. No architecture change. Text in, text out.
And it is what almost everything in production actually does.
7. Open world
A policy decides what to fetch.
Search agents, deep research.
The corpus is no longer fixed, the number of retrieval calls is no longer one, and the model is choosing.
- IRCoT, Self-RAG, ReAct → search-augmented RL (Search-R1)
- Report synthesis, citation faithfulness
Why I am teaching this
A hunch. Agent research keeps arriving at retrieval from every direction - memory, tools, context, grounding. I think the next few years of it are a retrieval problem wearing other names.
The group is turning. Our own work is moving towards AI plus retrieval, and it needs this literature rather than a summary of it.
So I am reading the field properly, now. Classical and current, in order, rather than in the fragments a paper deadline allows.
I am studying this with you
I am not teaching from a finished position. Some of this material is as new to me as it is to you.
Every week we take an idea from the classical work and ask what it does for a system built in 2026.
Bring the objection. If the honest answer is that nobody knows, that is a result, and it is where the projects come from.
What runs itself
| The deck | authored and served from a database, not a file. Once it is finished it is archived and does not change |
| The tutor | beside every deck, and /irai ask in Slack. It knows the syllabus, the policies and the lecture notes |
| The directory | who is in the course, and how to reach them |
| Your week | /irai me - your role, your deadline, whether your review is in |
| The project | your queries, your judgements, the leaderboard, from October |
Which half is which
| Do it with a model | Do it yourself |
|---|---|
| Decoding notation, filling in background | Choosing what is worth attacking |
| Drafting, reformatting, translating | Judging whether a passage is relevant |
| Code, plots, baselines, tooling | The ranking your group defends at 09:22 |
| Finding candidate supporting papers | Checking every number against the source |
| Rehearsing an argument before you make it | Changing your mind in front of people |
If a model doing it would remove the thing you came here to learn, do it yourself.
Why the argument stays human
At 09:22 you are not arguing with a model. You are arguing with four people who can be persuaded, who can be wrong in a way that teaches you, and who are marked on defending what your group wrote.
A model tends to concede when pushed, whether or not you are right. Agreement from it is weak evidence. Agreement from somebody who would rather disagree is not.
This is a method, not a course feature
The tutor that knows this material, the directory, the workbench these slides are written in - each of them built in an afternoon, because the mundane half was delegated.
You can build this for any subject you have to learn. Generate the scaffolding cheaply. Spend the attention you saved on the part that does not scale: reading closely, arguing with people, and deciding what is worth arguing about.
What we cover
Classical retrieval, quickly. The inverted index, BM25, evaluation, web search. Four weeks, and it is the toolkit everything later assumes.
Modern retrieval. Bi-encoders, approximate nearest neighbours, late interaction, retrieval under reasoning and ambiguity.
Retrieval joined to a language model. Inside the model - kNN-LM, RETRO, memory written at inference. Outside it - RAG, search agents, agent memory.
How the course runs
Mondays - a lecture on the method. Wednesdays - usually a structured discussion of a paper that uses it, in fixed groups of five.
Discussions start 30 September: the first three Wednesdays are lectures, and so are four later ones.
The only weekly deadline is Tuesday 18:00: a half-page written review.
The term, all at once
| Weeks | What |
|---|---|
| 31 Aug - 28 Sep | Lectures only. Inverted index, BM25, evaluation, web search, bi-encoders, ANN |
| 30 Sep | First discussion: BEIR. Judging protocol launches |
| 7 Oct | Workshop 1 (90 min). Build the test collection. Project kickoff |
| Oct - Nov | Late interaction, reasoning, ambiguity, attention-as-retrieval, memory |
| 11 Nov | Workshop 2 (90 min). Judgements freeze. Measure your modification |
| Nov - Dec | kNN-LM & RETRO, RAG, agents, agent memory, privacy |
| 16 Dec | Workshop 3 (90 min). Presentations. |
No class 21-23 Sep (Chuseok) or Mon 5 Oct.
Your group of five
Every Wednesday discussion runs in a group of five. The same five all term, from 30 September.
Choose them yourself. Name up to four people on the first-day survey by Mon 14 Sep.
A grouping proposal counts if everyone in it names everyone else. Agree on the group between yourselves first.
Name nobody and you are drawn at random. Every place left after 14 September is filled that way, and it is a perfectly good answer.
You have to find each other first. Who is in the course - every name, and how to reach them.
Five roles, rotating
Adversary - attack the experiments, find the missing baseline
Historian - what came before, and what this paper quietly discards
Replicator - do the numbers support the claims
Extender - propose the next paper and say what it would test
Applier - would you deploy this, at what index size, latency, cost
On the four newest papers, Replicator becomes Referee: would you accept it?
You choose your own supporting paper. Nobody assigns it.
The discussion hour
| Min | What |
|---|---|
| 0-5 | Framing, this week’s ranking criterion, three student questions |
| 5-22 | Groups. Table five points (5) → rank them (8) → file a written conclusion (4) |
| 22-40 | Open challenge. Groups defend against each other |
| 40-58 | Synthesis, corrections, placement on the spectrum |
| 58-60 | End-of-session questions, answered individually |
The member who answers is named when the group is called. There are no prepared presentations.
What is assessed
| Component | Weight |
|---|---|
| Weekly paper reviews | 45% |
| End-of-session questions | 20% |
| Contribution to discussion | 10% |
| Project | 25% |
The project: one BM25 baseline, one class-built test collection, one change. Predict its effect. Measure it. Explain the gap.
Beating the baseline is worth 1 point of 10.
Grades
A student who does all of the work competently lands in the middle of the distribution.
Ranked, cut into bands, roughly even across A+, A0, A-, B+, B0.
A+ - reviews that found something real, more than once
B+ - all the work done competently. Nothing missing, nothing exceptional
Where everything lives
Course page - irai.stai-lab.org - syllabus, schedule and recordings, rosters, Zoom link
Slack - join the workspace - announcements, weekly roster, questions, submissions. Ask here, not by email
Who is in the course - names, groups and addresses. Signed in only
Lecturer Seong Joon Oh
Tutors 추교욱, 이태훈
Mon & Wed 09:00-10:30. Sessions 9, 19, 29 run to 10:30. Online, every week.
Before you leave
1. Join the Slack workspace. Today.
2. Fill in the first-day survey - propose your group of five by Mon 14 Sep. Remaining places are filled at random.