The recording
You were last reading slide 1.
Coarse retrieval and reranking
Information Retrieval for AI
Session 6
16 September 2026
Seong Joon Oh - KAIST AI
Dates and BEIR preparation roles
BEIR preparation roles are assigned. Check your group and role on the course website. Sign in first.
No class on Monday 21 or Wednesday 23 September.
- Monday 28 September: lecture and a preparation reminder.
- Tuesday 29 September, 18:00: preparation note due.
- Wednesday 30 September, 09:00-10:00: BEIR discussion.
All times are KST. BEIR reading, preparation and discussion details are towards the end of this deck.
Coarse retrieval and reranking
Coarse retrieval and reranking
BM25 gives us an initial ranked list. A candidate is a document we keep for another scoring step.
Coarse retrieval: Precompute index prior to query. Given query, do only lightweight computation (e.g. BM25) to compute relevance.
Reranking: Compute relevance conditionally on query, with more expensive computation.
Sources: Sentence Transformers pipeline, Elastic hybrid search.
Recall @ K for coarse retrieval matters
Use the five documents from the evaluation example. a, b and c are relevant. The first search returns them in this order:
a, e, d, c, b
Keep only the first two: a, e. We have found one of the three relevant documents, so candidate recall is 1/3.
Reranking can change the order of a and e. It cannot put b or c in the results because neither was passed to it.
The reranker reads the query and document together
Query: dog bites man. Compare dog bites man with man bites dog. Their word counts match, but who bites whom changes.
A cross-encoder reads the query and a candidate together before it assigns a relevance score. It can use word order and how words in the two texts relate.
query + one candidate → model → relevance score
We train the model to prefer relevant candidates. Its ability to use word order does not ensure it learns this distinction; we need to test it.
Another option: an LLM reranker
A large language model can be prompted to score or order candidate documents. Its ranking quality and the time needed for the calls must also be measured.
Sources: Sentence Transformers pipeline, Sun et al., 2023, reranking agents.
Could we score every document with the reranker?
Suppose we search only the short passages in one manual.
We could send the query with every passage to the cross-encoder, then sort all its scores. No coarse stage would discard a passage first.
Would that be fast enough if many users searched at once?
Sources: Sentence Transformers pipeline; see the in-document search example.
Coarse retrieval with a bi-encoder
From term-count vectors to learned vectors
Session 3 scored a document by summing a weight over the query terms, then compared the resulting vectors by cosine.
TF-IDF:
BM25:
Both are sums of a product over terms, so both are dot products
| Score | Document coordinate | Query coordinate |
|---|---|---|
| TF-IDF | 1 for each query term | |
| BM25 |
The document coordinate uses only d and collection statistics (avgdl, and idf where we keep it on the document side), so it can be computed before the query arrives. It is nonzero only for terms occurring in d, which is what the posting lists store. Cosine divides the dot product by the two vector lengths. Repeated query terms take a further factor in the full BM25 form, and that factor sits on the query side.
We keep the dot product and replace the one-coordinate-per-vocabulary-term rule with coordinates a trained model computes from the text. The next question is what training should make close.
What should receive a high score?
Session 5 defined relevance from the user need before calculating a metric. Training needs the same decision.
- If the task is paraphrase search, “fix a flat bike tire” is a good match for “repair a punctured bicycle tyre”.
- If the task is question answering, a good passage should answer the question; repeating the question is insufficient.
We label examples according to that task, then train the score to prefer the labelled passages.
Sources: sentence-bert, dpr.
The vocabulary mismatch becomes a training question
Session 4’s car repair / automobile maintenance example showed a limit of word matching.
Here is a second constructed pair:
- “repair a punctured bicycle tyre”
- “fix a flat bike tire”
Can we train vectors whose similarity stays high across these word changes? We will check this pair in the measured demo, then test changes that should alter the meaning.
Sources: sentence-bert; constructed wording example; no retrieval result is claimed.
Bi-encoder architecture
An encoder learns the numbers in the text vector
With TF-IDF, each coordinate has a fixed vocabulary term. A learned encoder instead computes the vector’s numbers from the text using trained parameters.
A bi-encoder processes the query and passage separately. Their vectors meet only at the similarity score.
BERT is one model we can use inside the encoder. It first produces a vector for each token, a word or piece of a word. That vector depends on the surrounding text. Next we need one vector for the whole passage.
Sources: Reimers and Gurevych, Sentence-BERT (2019), Karpukhin et al., Dense Passage Retrieval (2020); SBERT section 3; DPR section 3.1.
BERT returns one vector per input token
N tokens in → N contextual vectors out. Each vector has the same dimension d.
The initial vectors add token, position and segment embeddings. BERT lets each position use context from both sides.
Output [CLS] and [SEP]. The vectors share a size, not their values. Pooling comes next.
Sources: Devlin et al., BERT (2019), section 3 and Figures 1-2; BertModel output shape. Four-token teaching example; single unpadded input.
Turn several token vectors into one passage vector
We need one vector for each searchable passage. Pooling is the rule that produces it from the token vectors.
- Mean pooling: average the vectors coordinate by coordinate. If a toy input has two vectors [2, 0] and [0, 2], their mean is [1, 1].
- [CLS] pooling: take the final vector at the special
[CLS]token that BERT places at the start of the input.
Original Sentence-BERT (SBERT) defaults to mean pooling. Dense Passage Retrieval (DPR) uses [CLS].
If dummy padding tokens were added to equalise input lengths, exclude them from the mean.
Sources: Reimers and Gurevych, Sentence-BERT (2019), Karpukhin et al., Dense Passage Retrieval (2020), Model card: all-MiniLM-L6-v2; SBERT section 3; DPR section 3.1; model-card pooling example.
Does the query use the same encoder as the passage?
Both branches produce vectors separately. We can still choose whether they use the same trained parameters.
SBERT shares its parameters between the two sentences. DPR has separate question and passage encoders.
Thus “two towers” describes the two encoding computations. It does not tell us whether their parameters are shared.
Sources: sentence-bert, dpr; SBERT Figure 1; DPR section 3.1.
Compare the query vector with the passage vector
Let q be the query vector and p the passage vector. Their vectors must have the same number of coordinates.
Dot product: multiply corresponding coordinates and add them.
Cosine: divide the dot product by the lengths of both vectors. The symbol
For nonzero vectors scaled to length 1, dot product equals cosine. DPR uses dot product; use the model’s scoring rule when searching its stored vectors.
Sources: Karpukhin et al., Dense Passage Retrieval (2020); section 3.1 and Appendix B.
Precompute document work, as we did with the term index
The inverted index let us prepare document information before a query arrived. Stored passage vectors serve the same purpose for a bi-encoder.
At query time, compute the query vector and search the stored passage vectors. Fetch the selected text by ID. If the passage encoder changes, its stored vectors may need to be rebuilt.
Sources: dpr; section 3.1.
Quick quiz: how many vectors survive pooling?
An encoder receives 6 tokens, including special tokens. Each token vector has 4 coordinates. Mean pooling averages across token positions. The passage, preprocessing and encoder stay unchanged for the next query.
Which statement gives the output shapes and passage-vector reuse correctly?
Sign in with your course account to choose. The lecturer will reveal responses and the answer.
Constructed teaching dimensions, not a model-size claim.
Bi-encoder training
Training needs labels that say which texts should match
A sentence-pair task may ask whether two sentences express the same meaning, or how similar their meanings are. Original Sentence-BERT studies several objectives for such comparisons.
DPR instead trains a question against passages that should answer it.
For either task, a training label says which match to prefer. The loss turns those labels and the current scores into a number that training tries to reduce.
Next we will use a question, an answering passage and a distracting passage to calculate that loss.
Sources: Reimers and Gurevych, Sentence-BERT (2019), Karpukhin et al., Dense Passage Retrieval (2020); SBERT section 3; DPR section 3.2.
Give the model an answer and a distractor
These are constructed training texts.
| Training input | Text |
|---|---|
| Question | What does the red battery indicator mean? |
| Positive: the passage to prefer | A red battery indicator means the charge is low. |
| Negative: the passage to score lower | The device is available in red and blue. |
“Positive” and “negative” describe the training labels. The negative shares the word “red”, but does not answer the question. We want the model to give the positive the higher score.
Sources: Karpukhin et al., Dense Passage Retrieval (2020); section 3.2; texts are constructed.
Where did DPR’s positive passages come from?
For three question-answer datasets, TREC, WebQuestions and TriviaQA, the DPR paper uses this procedure:
- Search with BM25 and inspect its top 100 passages.
- Among passages that contain the recorded answer text, choose the highest-ranked one as the positive.
- If none contains that answer, discard the question.
This makes training examples from existing questions and answers. It also means BM25’s shortlist and the answer-text check affect which examples reach training.
Sources: Karpukhin et al., Dense Passage Retrieval (2020); section 4.2; this is not the positive-construction rule for every DPR dataset.
Train the relevant passage to outrank selected alternatives
For one query, take a labelled relevant passage
The training objective rewards
Next we will turn that preference into a numerical loss.
Sources: dpr; section 3.2.
Reward the positive for a larger share of the scores
For query q, let C contain the positive passage
- Replace each score s with
, a positive weight. - Divide the positive’s weight by the sum of all weights. This is softmax:
- Take its negative natural logarithm:
.
Training minimises L by changing the encoder parameters. Giving the positive a larger share lowers L. This probability compares passages in C; it is not a calibrated probability of relevance.
Sources: Karpukhin et al., Dense Passage Retrieval (2020); equation 2.
Reuse the other examples’ passages as negatives
A batch is a group of examples processed in one training step. Suppose it contains matched pairs (
For
Let Q and P be matrices whose rows contain the query and passage vectors. Compute
The diagonal entries pair each query with its positive. This reuses each encoded passage across several comparisons. The next slide shows the numbers.
Sources: Karpukhin et al., Dense Passage Retrieval (2020); section 3.2, in-batch negatives.
Read one row as one question’s ranking problem
These are constructed dot products, not measured model outputs. Question
| Query / passage | |||
|---|---|---|---|
| 3 | 2 | 0 | |
| 1 | 3 | 0 | |
| 0 | 2 | 3 |
For
Which question has the smallest loss? Compare how far its positive score is above the competing scores. We will calculate all three losses next.
Sources: Karpukhin et al., Dense Passage Retrieval (2020); constructed dot products; no checkpoint produced this matrix.
Quick quiz: the positive scores tie; do the losses?
Both questions give their positive passage a score of 3. For q₁, the negatives score 2 and 0. For q₂, the negatives score 1 and 0. Apply softmax over each question’s candidates, then take the negative log of the positive’s share.
Which question has the smaller loss?
Sign in with your course account to choose. The lecturer will reveal responses and the answer.
Constructed scores; natural-log arithmetic checked locally.
A negative is selected for a particular question
DPR compares these ways to obtain a passage labelled negative:
| Name in the paper | How it is chosen |
|---|---|
| Random | Sample a passage from the collection |
| BM25 | Take a high-ranked BM25 passage that lacks the answer text |
| Gold | Use another question’s positive passage |
For example, a passage about a blue indicator could answer a different question while distracting from our red-indicator question.
A hard negative is a passage that scores highly for this query despite its negative label. It competes with the positive.
Sources: Karpukhin et al., Dense Passage Retrieval (2020); section 3.2.
Do not penalise another passage that answers the question
Our positive says: “A red battery indicator means the charge is low.”
Suppose another passage says: “When the battery runs low, the indicator turns red.” It also answers the question.
If training labels that passage negative, the model is asked to push down a useful result. This is a false negative.
One response is to exclude a suspect negative from that training comparison. Later we will see methods that do this with a mask, a rule that sets the unwanted loss contribution to zero.
Sources: Karpukhin et al., Dense Passage Retrieval (2020), Gemini Embedding technical report (2025); constructed task; Gemini section 3.2 discusses false-negative handling.
Bi-encoder analysis
Define the similarity task
For this demonstration, “same meaning” must preserve:
- who did what to whom;
- whether an event happened;
- which entity or repair need is named.
A shared topic is not sufficient.
Sources: Sentence Transformers similarity.
Which candidate should the similarity score prefer?
| ID | Sentence |
|---|---|
| 1 | The dog chased the cat. |
| 2 | The cat was chased by the dog. |
| 3 | The cat chased the dog. |
| 4 | The dog did not chase the cat. |
| 5 | The dog chased the mouse. |
| 6 | How can I repair a punctured bicycle tyre? |
| 7 | How do I fix a flat bike tire? |
| 8 | How do I replace a bicycle chain? |
| 9 | Paris is the capital of France. |
| 10 | Berlin is the capital of Germany. |
For sentence 1, which of candidates 2, 3, 4, 5 preserves its meaning? That candidate should receive the highest score for this task.
Predict whether the model will agree before we reveal its scores.
Sources: Model card: all-MiniLM-L6-v2; ten constructed inputs; compare the tyre-repair pair 6 and 7 too.
The model gives the wrong pair a higher score
We encoded the ten sentences with all-MiniLM-L6-v2 and calculated cosine between every pair. A larger score means this model places the two vectors closer in direction.
| Reference and candidate | Same meaning? | Measured cosine |
|---|---|---|
| Dog chased cat / cat was chased by dog | Yes | 0.9632 |
| Dog chased cat / cat chased dog | No | 0.9913 |
The reversed roles receive the higher score. A high cosine does not ensure that the sentences mean the same thing.
Full ten-sentence matrix and run details
Rows and columns use the sentence IDs on the previous slide. A cell compares its row sentence with its column sentence. Diagonal cells compare a sentence with itself. Values are rounded for display.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1.00 | 0.96 | 0.99 | 0.73 | 0.63 | 0.08 | 0.06 | 0.03 | -0.01 | 0.01 |
| 2 | 0.96 | 1.00 | 0.97 | 0.75 | 0.59 | 0.05 | 0.02 | 0.02 | -0.04 | 0.00 |
| 3 | 0.99 | 0.97 | 1.00 | 0.72 | 0.61 | 0.07 | 0.05 | 0.03 | -0.03 | 0.01 |
| 4 | 0.73 | 0.75 | 0.72 | 1.00 | 0.46 | 0.03 | 0.02 | 0.02 | 0.02 | 0.01 |
| 5 | 0.63 | 0.59 | 0.61 | 0.46 | 1.00 | 0.00 | -0.04 | 0.04 | 0.07 | 0.10 |
| 6 | 0.08 | 0.05 | 0.07 | 0.03 | 0.00 | 1.00 | 0.67 | 0.37 | 0.04 | -0.05 |
| 7 | 0.06 | 0.02 | 0.05 | 0.02 | -0.04 | 0.67 | 1.00 | 0.45 | 0.01 | -0.05 |
| 8 | 0.03 | 0.02 | 0.03 | 0.02 | 0.04 | 0.37 | 0.45 | 1.00 | -0.03 | -0.05 |
| 9 | -0.01 | -0.04 | -0.03 | 0.02 | 0.07 | 0.04 | 0.01 | -0.03 | 1.00 | 0.35 |
| 10 | 0.01 | 0.00 | 0.01 | 0.01 | 0.10 | -0.05 | -0.05 | -0.05 | 0.35 | 1.00 |
CPU run, 15 September 2026. Float32 vectors scaled to unit length. Model revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. The saved output retains full precision.
Sources: Model card: all-MiniLM-L6-v2, Sentence Transformers similarity.
The model confuses who chased whom
Sentence 1: The dog chased the cat.
| Candidate | Meaning preserved? | Measured cosine |
|---|---|---|
| The cat was chased by the dog. | Yes | 0.9632 |
| The cat chased the dog. | No | 0.9913 |
The second score is higher, although the cat and dog swap who chases whom. Recall session 4’s dog bites man / man bites dog example: a learned vector score also needs to be tested on this distinction.
We computed exact pair similarities. Every displayed pair was compared directly. Approximate search did not cause this error. This observation concerns the model and sentences in this run.
Sources: Model card: all-MiniLM-L6-v2, Sentence Transformers similarity.
Reranking with a cross-encoder
A cross-encoder computes one score for the whole pair
For a BERT-based cross-encoder, the input contains the query followed by the candidate passage, separated by special tokens.
query + passage → joint token representations → learned scoring layer → one score
Attention can connect words in the query with words in the passage. A small learned output layer turns the resulting representation into a relevance score.
For the red-warning question, the model can compare “red indicator” with the passage’s explanation of a low battery. It must learn to prefer that explanation over a passage about a red casing.
Sources: Hugging Face BERT input format, Sentence Transformers cross-encoder training.
Cross-encoder architecture: two texts in, one score out
The query and one candidate form one token sequence. BERT lets their tokens use each other’s context.
BERT still outputs one vector per token. The scoring layer uses the final [CLS] vector to produce one number for this pair. Repeat for each candidate, then sort by score.
Shown: a BERT-based cross-encoder with a scalar scoring head. Sources: Nogueira and Cho, Passage Re-ranking with BERT, method; BERT, section 3. A sigmoid can map the score to 0–1.
Why use separate vectors first and joint scoring later?
A bi-encoder lets us reuse each passage vector across queries. The query and passage meet at the final vector comparison.
A cross-encoder reads both texts together before it gives a score. Its passage computation depends on the query, so we cannot store one such passage vector for every future query.
This explains the two stages at the start of the lecture: search precomputed information across the collection, then spend query-specific model work on the shortlist.
Sources: sentence-bert, Sentence Transformers pipeline.
Train the cross-encoder on labelled query-passage pairs
Reuse the query “What does the red battery indicator mean?”
| Passage | Label |
|---|---|
| A red battery indicator means the charge is low. | 1: relevant |
| The device is available in red and blue. | 0: not relevant |
One training choice is binary cross-entropy. Convert each pair’s score to p between 0 and 1. A positive pair incurs loss −log p; a negative pair incurs −log(1−p). Training changes the model parameters to reduce these losses.
Here each pair receives its own label. This differs from the earlier softmax comparison across a query’s candidate passages. Other cross-encoder losses can compare candidates jointly.
Use the new scores to reorder the same shortlist
Suppose the bi-encoder returns candidates A, B, C. Score each query-passage pair with the cross-encoder.
| Candidate | Constructed cross-encoder score |
|---|---|
| A | −0.5 |
| B | 2.0 |
| C | 1.0 |
Sort by these scores: B, C, A. The scores are classroom inputs, not measured model outputs. A reranker can return raw scores outside 0 to 1; ranking only needs their order.
For K candidates, we compute K pair scores. Compare the final ranking with the bi-encoder’s order on the same judged queries, and record reranking time. Documents absent from the shortlist remain unavailable.
Sources: Sentence Transformers cross-encoder usage, Sentence Transformers retrieve and rerank.
Quick quiz: what can a cross-encoder reuse?
A cross-encoder has scored query A with passage P and saved the final [CLS] vector. Query B now retrieves K passages, including P.
What work is needed to score query B’s shortlist?
Sign in with your course account to choose. The lecturer will reveal responses and the answer.
BEIR discussion
BEIR discussion: Generalisation of dense retrievers
Wednesday 30 September, 09:00-10:00 KST
Central questions:
What does BEIR let us conclude about the OOD generalisation of dense retrieval? How relevant is that conclusion in 2026?
| Date/time | What to do | Graded |
|---|---|---|
| Before the session | Read BEIR. | |
| Before the session | Read an additional supporting paper for your role. | |
| 29 Sep, 18:00 | Submit your preparation note. | ✓ |
| 30 Sep, 09:00-09:05 | Receive the live scenario. | |
| 30 Sep, 09:05-09:30 | Discuss with your group. | |
| 30 Sep, 09:30-09:40 | Selected students answer follow-up questions. | ✓ |
| 30 Sep, 09:45 | Submit your closing paragraph. | ✓ |
| 30 Sep | Submit the conduct questionnaire. |
Your group conclusion and closing paragraph must apply the live scenario revealed in the lecture.
Roles and supporting papers
Check your assigned role and group. Each student prepares for one role; four-person groups leave one role unassigned.
| Preparation role | What to examine | Suggested reading |
|---|---|---|
| Adversary | The strongest objection | GTR |
| Historian | What BEIR changes from earlier evaluation | MS MARCO |
| Replicator | Whether reported results reproduce | Brewing BEIR |
| Extender | What later evaluation adds | MMTEB |
| Applier | Retrieval quality under resource constraints | ColBERTv2 |
Choose one relevant paper yourself. Declare it in your preparation note. Duplicate choices are fine.
Your role guides preparation. This doesn’t necessarily mean you have to stick to the position during discussion. For example, an adversary may eventually conclude that a criticism against the paper fails.
Preparation note - graded individually
Due Tuesday 29 September, 18:00 KST. Open the BEIR preparation form. Autosave keeps a draft; press Submit preparation note.
Graded: your main argument, reasoning and evidence, and remaining question with its explanation. The question is part of the note, with no separate mark.
| Field | What to write |
|---|---|
| Role | Your preparation role |
| Supporting paper name | Authors, year, title, and a link or DOI |
| Main argument | One sentence |
| Reasoning and evidence | 2-3 paragraphs: explain your reasoning and cite sections, tables or factual observations |
| Remaining question & question for peers | One paragraph: state the question and explain why the reading has not already answered it |
| AI use | If used: prompts, outputs, what was wrong or missing, and how your writing differs. No length limit. Otherwise, write None |
Example preparation note
Role: Adversary.
Supporting paper: Ni et al. (2022), Large Dual Encoders Are Generalizable Retrievers.
Main argument: GTR challenges the interpretation that BEIR reveals an inherent limitation of dense dual encoders.
Reasoning and evidence
GTR’s introduction questions whether limited interaction between query and document representations explains weak transfer. Its scaling experiments retain one vector for each query and document and a single dot-product comparison, but increase encoder size.
Table 3 reports average nDCG@10, excluding MS MARCO, of 0.416 for GTR-Base, 0.458 for GTR-XXL and 0.423 for BM25. These results weaken a general claim that dual encoders must transfer poorly. However, GTR-XXL still scores below BM25 on TREC-COVID. Its average advantage does not establish that it is the best choice for every new domain.
Source: GTR, introduction and Table 3, published EMNLP version.
Remaining question & question for peers
Should GTR’s average improvement make it our default starting point for a new domain? The table contains both an overall improvement and losses on individual datasets, so it does not settle that decision. What additional evidence would the group require before choosing GTR?
AI use: None in this illustrative submission.
The question gives peers something to reason about. Compare it with Does GTR’s size affect its average score?, which the experiments already answer.
The discussion hour
| Time on 30 September, KST | Activity |
|---|---|
| 09:00-09:05 | Lecturer reveals one scenario to the whole class |
| 09:05-09:30 | Apply your reading to the scenario; manage the discussion yourselves |
| 09:30-09:40 | One or two students per group answer short graded follow-up questions |
| 09:40-09:45 | Write your graded individual closing paragraph |
| 09:45-09:57 | Lecturer synthesis and remaining questions |
| 09:57-10:00 | Private questionnaire |
Graded individually: preparation note, closing paragraph and the selected follow-up responses. Contribution to discussion is 10% of the course mark.
Group outcome: for the scenario, explain your conclusion, the fact that matters most to it, and the strongest objection. If you disagree, explain where the disagreement remains.
Give everyone room to contribute a reason or piece of evidence. No timed opening speeches or group report. Finish group discussion at 09:30. Follow-up questions end at 09:40.
Contribution to discussion - 10%
09:30-09:40: the lecturer draws one or two students per group for a short question and follow-up. The slot stays ten minutes.
The draw favours students with fewer previous calls, with random ties. You do not need to speak to the whole class every session; we aim for similar numbers of turns across the course.
| Score | What your response does |
|---|---|
| 2 | Answers with a reason; addresses the follow-up or acknowledges a limitation |
| 1 | Answers, but repeats the conclusion without addressing the challenge |
| 0 | Cannot answer the question |
We assess reasoning, not fluency or agreement with the lecturer.
Your mean score out of two becomes a mark out of ten. Not selected or not reached means no score for that session.
Closing paragraph - graded individually
Submit about 100 words in the closing paragraph section after discussion and before the lecturer synthesis. It opens around 09:40. Work without AI.
What do you conclude for the live scenario, which fact most affects your decision, and how did another student’s argument affect your reasoning?
Your reasoning is assessed. Keeping or changing your position earns or loses no marks in itself. Explain how the live case and your peer’s argument support or limit your conclusion.
English, AI and the closing questionnaire
Use English for preparation, discussion and written responses.
| When | AI rule |
|---|---|
| Preparation | Allowed, with disclosure in the preparation note |
| Discussion session | Prohibited, including the individual closing paragraph |
Preparation AI use is optional; this replaces the earlier mandatory model check.
The questionnaire is a required conduct report, not a graded assignment. At 09:57-10:00, open the conduct questionnaire and answer privately about each other group member:
- English: throughout / sometimes another language / could not assess.
- AI: no use observed / use directly observed / could not assess.
Other members may report any AI use they observe during the session. If you report a concern, describe what you saw or heard in one sentence. The lecturer reviews specific reports; peers do not award or deduct marks. The questionnaire covers visible classroom conduct.