The text below records the earlier format and scoring rules. It is not the current marking rubric. Use the current discussion policy above.

Choosing your supporting paper

Policy, from the 2026 revision. Supporting papers are not assigned. Each week you hold one role, and you choose the paper that serves it. Recommendations exist for every role on every assigned paper, and taking one is a perfectly good answer - but the choice, and the defence of it, is yours.

This document is exhaustive on purpose. If a case is not covered here, the resolution is in “Unlisted cases” at the foot.

1. The rules that apply to every role

R1. Declare it. Your review opens with one line, before the four fields:

Supporting paper: Author(s), year, title, venue. URL or DOI. Why it fits the role: one sentence.

A review with no declaration line is marked as if no supporting paper was read.

R2. Choose by Tuesday 18:00. There is no approval step and no earlier deadline. You declare it in the review; that is the moment of choice.

R3. It must be checkable. Anyone marking must be able to open it. A DOI, an arXiv id, or a stable URL. Paywalled work is fine if KAIST has access; say so.

R4. One paper. Not two, not a survey plus a paper. If you read more, declare the one you are relying on.

R5. Not the anchor, and not another anchor. The seven assigned papers are excluded as supporting papers, in any role, in any week. They are already read by everyone.

R6. No reuse. A paper you declared in an earlier week cannot be declared again, in any role. Across seven discussions you will have read seven distinct supporting papers.

R7. Duplicates between students are fine. Two people may land on the same paper. There is no coordination requirement and no penalty. You are not competing for papers.

R8. Papers only, with two exceptions. A supporting paper is a research paper, a book chapter, or a technical report. Blog posts, documentation, code repositories and review threads are admissible only for Applier and Referee, where the artefact type is the point of the role.

R9. English, or translate. If the passage you rely on is not in English, quote it and give your translation.

R10. Authorship is not a bar. A paper by the anchor’s own authors is admissible if it independently meets the role test. A later paper by the same group is often the correct Extender choice.

R11. If nothing fits, take the recommendation and say why. Declare the recommended paper, and add one sentence on what you searched for and why the search failed. This scores 2. It does not score 0. An honest failed search is a result; a silent substitution is not.

2. The role tests

Each role has an admissibility test - an objective question with a yes or no answer - and a set of actions that the review must contain for full marks.

A paper that fails the admissibility test caps the review at 1, however good the writing. A paper that passes but whose actions are missing caps it at 2.

Adversary

Test. Does this paper make a claim which, if true, weakens the anchor’s central claim? You must be able to complete this sentence from the paper’s own content: “If [supporting paper] is right, then the anchor’s claim that [X] is overstated, because [Y].”

Admissible. A paper reporting a contrary result. A paper showing the anchor’s baseline was weak or its protocol confounded. A paper whose method achieves the anchor’s outcome without the anchor’s mechanism. A methodological critique of the family of work the anchor belongs to.

Inadmissible. A paper that merely does something different. A paper that is worse than the anchor at the same task with no argument about why the comparison is unfair. A later paper that improves on the anchor without contesting it - that is the Extender’s.

Actions for full marks.

1. Name the anchor’s specific claim under attack, quoting it or citing its table.
2. State the contrary evidence, with the number or figure it rests on.
3. Say what would settle the disagreement - the experiment, the missing baseline, the measurement nobody ran.
4. Say honestly whether the anchor survives, and on what conditions.

Historian

Test. Does this paper predate the anchor, and does the anchor build on it or react to it? A citation in the anchor is sufficient evidence. Absent a citation, you must argue the lineage.

Admissible. A cited predecessor. The paper that introduced the mechanism the anchor modifies. The paper that framed the problem the anchor attacks. An older paper making substantially the same argument, where the point is that the anchor rediscovered it.

Inadmissible. Anything published after the anchor. A contemporary paper with no lineage relationship. A general survey.

Actions for full marks.

1. Name what the anchor took from it, concretely - a formulation, a dataset, a framing, an assumption.
2. Name what the anchor discarded, which is the harder half and the point of the role.
3. Say whether discarding it was justified, and what was lost if not.

Replicator

This role becomes the Referee on sessions 15, 21, 25 and 27. See below.

Test. Does this paper report numbers for the anchor’s method that the anchor’s own authors did not produce?

Admissible. A reproduction study. A benchmark paper that re-evaluates the anchor’s method under a common protocol. A later paper whose baseline table includes the anchor. An independent reimplementation with results.

Inadmissible. The anchor’s own follow-up reporting its own numbers. A paper that cites the anchor’s numbers without re-running anything. A paper about reproducibility in general with no numbers for this method.

Actions for full marks.

1. Pick one specific number from the anchor.
2. Give its counterpart from your paper.
3. State the size and direction of the discrepancy.
4. Give the most likely cause - different data, different tuning, different metric, different hardware - and say how you would test that.

Referee

On the six recent assigned papers, the Replicator becomes the Referee, and this is the one role where the document is fixed, because a paper has at most one public review thread. You do not choose it. You are told which thread to read, or told that none exists.

Actions for full marks.

1. Before opening the thread, write your own verdict on the anchor: accept, reject or major revision, with three reasons and a score out of 10. Timestamp it by pasting it into your review first.
2. Read the thread, where one exists.
3. Name one thing the reviewers caught that you missed, and one thing you caught that they missed. If you genuinely find neither, say so and defend it.
4. Where no thread exists, your review is the deliverable, and step 3 is replaced by: name the objection you expect the authors would find hardest.

Extender

Test. Does this paper postdate the anchor and build on it? A citation of the anchor is sufficient evidence.

Admissible. A successor method. An application of the anchor’s mechanism to a new setting. A paper that generalises it. A paper that fails to extend it, where the failure is informative.

Inadmissible. Anything published before the anchor. A paper that merely cites the anchor in passing without using anything from it.

Actions for full marks.

1. Name what the successor kept and what it changed.
2. Name the experiment that should come next, which neither paper has run.
3. Say what that experiment would settle - what result would change your mind about the anchor, in which direction.

A proposal that could be written without having read either paper - “test it at larger scale”, “try more datasets” - scores 1.

Applier

Test. Does this artefact contain at least one quantitative resource number for a real deployment - index size, latency, throughput, memory, or cost?

Admissible. A systems paper. An engineering write-up or blog post from a team that runs the thing. Library or engine documentation with benchmark tables. A toolkit paper. The anchor’s own resource table only if you combine it with an outside source.

Inadmissible. An artefact with no numbers. A pricing page with no workload attached.

Actions for full marks.

1. State the deployment scale you are costing - number of documents, queries per second, or corpus size. Pick one and name it.
2. Give an estimate at that scale, with the arithmetic shown.
3. Name the binding constraint - what runs out first, and at what point.
4. Answer plainly: would you deploy this, and if not, what would have to change?

Every number must be reproducible from your review. This is the role where the course’s bright line bites hardest.

3. How the choice is marked

The paper choice does not carry its own score. It gates the review score you already know:

The recommendation is never penalised. Taking the recommended paper and doing the actions well scores exactly what an independent find scores. The course pays for what you do with the paper, not for the effort of locating it.

4. Recommendations

A recommended paper exists for every role on every assigned paper. They are published with the roster two weeks ahead, and they sit in the reading list on the public page. They are a floor, not a ceiling, and they carry no advantage and no penalty.

5. Unlisted cases

A case not covered here is resolved in the student’s favour for that week, and this document is amended before the next session. Raise it on Slack rather than by email, so the answer reaches everyone at once.

Amendments are dated and listed at the foot of this file. A rule added mid-term never applies retroactively to a review already submitted.

Amendments

• 2026-08-23. First version, replacing pre-assigned supporting papers.