Your answer key is the product: building a gold set you can trust

Ninety-four percent, but whose key?

A team builds a system that reads supplier emails and flags shipment delays. On their test set it scores 94%. The number goes on a slide, and the first question anyone careful should ask is not about the model at all. Who wrote the answer key, and why should anyone trust it?

Every accuracy figure compares two columns: what the system said, and what the key says is right. Teams pour effort into the first column and very little into the second. If the key is wrong, inconsistent, or quietly derived from the model itself, the 94% measures something, just not what the slide claims.

The tempting shortcut is familiar: let the model label a big batch, have someone skim its answers, and fix whatever looks wrong. The course puts numbers on why that fails.

Take 1,000 model labels, of which 80 are wrong. A reviewer skimming the suggestions catches 31% of the errors, so 80 x 0.31 = 24.8, about 25 labels get fixed. The rest are approved, which gives an approval rate of 975 out of 1,000, or 97.5%. The batch looks clean. But 80 minus 25 leaves 55 wrong labels inside those 975 approved items, and 55 / 975 is 5.6%. That error now lives in the answer key, and every model you test against it is graded partly on agreement with the old model's mistakes.

The same pattern appears when a review process reports 99% approval and someone finally runs a blind audit. In the course example the audit found 17 of 200 items wrong, about 1 in 12 (17 / 200 is 8.5%), with a 95% interval of 5.4% to 13.2%. Even the low end is far above the 1% that 99% approval seems to promise.

So here is the thesis. A model is only as measurable as its gold set: the examples with known correct answers that you measure the system against. Gold does not mean guaranteed truth. It means the answers were established carefully, by a process you can describe, and stay open to correction. That process is the real product: blind, double labelled, stratified, versioned and split by time, with kappa telling you whether the labels themselves can be trusted.

Anchoring: a visible answer bends judgment

Two tasters try the same glass. One is told it is something special and finds layers of flavour. The other is told nothing and finds it pleasant. Same glass, different answer. Nobody lied. The first taster's judgment bent toward the suggestion before they noticed.

Labellers work the same way. Show a labeller the model's answer first and the task quietly changes. Instead of reading the email and deciding whether it describes a delay, they start checking whether the suggestion looks plausible. Those are different tasks with different error rates. The 31% catch rate above is what checking looks like in practice: most wrong answers look plausible, which is exactly why the model produced them.

The harm compounds. If the gold set inherits the model's mistakes, the model looks better on it than it is, and a later model that fixes those mistakes looks worse. A score built this way partly measures how persuasive the model was. I would not put that number in front of anyone making a decision.

The same thing happens between people. If one labeller can see the other's answer, agreement climbs for the wrong reason: someone is partly copying. The fix in both cases is blind independence. Each labeller sees the input, the context and the guidelines. They never see the model's output, and they never see each other's call until both labels are recorded.

This rule is easy to break by accident. A labelling tool pre-fills the model's prediction. A spreadsheet hides the prediction column instead of removing it. A second labeller works from a copy of the first one's file. Each turns two independent judgments into one judgment and an echo, and nothing in the final numbers shows it.

The six rules of a gold set

The module gives six rules. None is exotic. What matters is that all six hold at once, because each closes a different way for the key to drift from the truth.

Rule 1: label blind. Labellers see the input, the context and the guidelines, never the model's answer.

Rule 2: write guidelines, version them, and store the version on every label. Each label needs a definition, examples and boundary cases, and the guideline's version is recorded on every label row. This sounds like ceremony until the score moves. Suppose it falls next month. Did the model get worse, or did the definition of "delay" get stricter? Without the version on each label, nobody can tell. With it, you can also answer the next question: when a definition sharpens, which old labels need a second look?

Rule 3: two labellers, then a tie-breaker. Each item gets two independent labels. Where they disagree, the guideline owner decides. A disagreement usually points at a gap in the guidelines, not a careless labeller. The effort is smaller than most teams assume. Two labellers on 400 items at one minute each is 400 x 2 x 1 = 800 minutes, about 13 hours, plus some tie-breaking. That is less than a week of one person's time, for a measurement you will use every week for a year.

Rule 4: stratify. Random samples are full of easy, common cases. If an event happens once in 2,000 emails and you sample 50 at random, the chance of seeing it even once is 1 minus (1999 / 2000) raised to the 50th power, which is 0.0247, about 2.5%. So plan the mix on purpose: every category, every language, every major supplier, plus deliberately hard cases. The lessons name several: one email covering several orders, very short replies, long quoted threads, a changed policy, the wrong customer's account, a request sent twice. Over-sample the rare classes, or you will end up with three examples and no measurement.

Stratifying has one catch: the mix no longer looks like real traffic. Weight each group back to how often it really appears. In the course example, the common group is 80% of real traffic and scores 99%, and the hard group is 20% of traffic and scores 80%. The weighted estimate is 0.8 x 99 + 0.2 x 80 = 79.2 + 16 = 95.2%. The raw average over the set was 89.5%, which is what you get when the two groups are equally represented: (99 + 80) / 2 = 89.5. Report the weighted figure and the per-group figures together.

Rule 5: split by time and source. Keep a development set, a test set from later months, and a holdout you rarely touch, with each supplier on one side. It gets its own section below.

Rule 6: find your own label mistakes. Even careful people mislabel a few percent of items. Use the model's probabilities to point at the labels most likely to be wrong, such as items where the model is confident and the key disagrees, and re-check those first against the guideline and the original email. The model points; a person decides.

Rule 6 also covers other sources of truth. Outcome labels, such as whether the goods arrived on the promised date, are slow but need no extra labelling effort and are independent of labellers and model. Synthetic examples can fill rare gaps in development, never in the test set.

Kappa from a 2x2 table, and the rough 0.6, 0.7, 0.8 bands

Rule 3 gives you two labels per item. Do the labellers agree enough for their labels to count as a key? Raw agreement alone misleads.

Start with the second lesson's example. Two people label the same 100 supplier emails as delay or not delay, and they match on 85. Is that good? If both called nearly everything "not a delay", they would match constantly without reading closely.

Put the 100 emails in a 2x2 table, one labeller across the top and the other down the side, and look at the totals along each edge. One person said "delay" 45 times; the other said it 50 times. Those totals are called marginals: how often each rater uses each label overall. A marginal is a habit, not evidence of being right.

Now ask how often the two would match if each kept those habits but labelled independently. Both say delay with probability 0.45 x 0.50 = 0.225. Both say not delay with probability 0.55 x 0.50 = 0.275. Together that is 0.50: half the time, they would match by habit alone. That is chance-expected agreement.

Picture a thermometer. The bottom is that chance line at 50%, and the top is perfect agreement at 100%. The pair reached 85%, which is 35 points above the line out of a possible 50. 0.35 / 0.50 = 0.70. They got seven tenths of the way up. That share is kappa: agreement gained above chance, as a fraction of the most that could be gained.

The slide version uses a different table and lands on the same value, which makes it a good check for your own spreadsheet.

Observed agreement is (80 + 12) / 100 = 0.92. Chance agreement is 0.85 x 0.83 + 0.15 x 0.17 = 0.7055 + 0.0255 = 0.731. Kappa is (0.92 minus 0.731) / (1 minus 0.731) = 0.189 / 0.269 = 0.70. In counts: by luck alone they would agree on about 73 emails, they actually agreed on 92, so they gained 19 above luck out of a possible 27, and 19 / 27 is about 0.70.

The bands are rough conventions, not laws. Above 0.8 is strong agreement. Below 0.6, fix the definition before you blame the model, because the key is contradicting itself. Around 0.7, where both examples land, I treat the labels as usable for development and worth a disagreement meeting before they anchor a release decision.

Kappa has a trap of its own. When almost everything gets one label, the chance line climbs close to the top, and the same matches barely clear it. In the lesson, two tables each have 95 matches out of 100: with labels split about evenly, kappa is about nine tenths; with nearly everything one label, about a quarter.

The lab's worked case uses 1,000 outputs graded by people and by a model judge. People pass 960 and the judge passes 970. They agree on 950: 940 both pass and 10 both fail, with 20 items people passed and the judge failed, and 30 items people failed and the judge passed. Raw agreement is 95%. Chance agreement is 0.97 x 0.96 + 0.03 x 0.04 = 0.9312 + 0.0012 = 0.9324. Kappa is (0.95 minus 0.9324) / (1 minus 0.9324) = 0.0176 / 0.0676 = 0.26.

That is the prevalence paradox, and it is not a reason to fall back to raw agreement. Look at the failure cell. People failed 40 items. The judge also failed only 10 of them, so agreement on failures is 10 of 40, or 25%. Those rare failures are the cases you care about, and 95% agreement says almost nothing about whether the judge catches what people catch. A low kappa means inspect the whole table. It does not prove the definition failed, and it cannot be waved away. This is the same reason two models agreeing with each other is weak proof, which I wrote about in model consensus is not engineering evidence.

High kappa has a limit too. Give both labellers the same wrong rule, say "every email containing the word pending is a delay", and they agree on every email. Kappa hits 1. Then an independent audit reads the full emails against the agreed definition and finds many were conditional updates, not confirmed delays. Kappa adjusts agreement for label habits. It says nothing about truth. A high kappa does not let you skip the audit.

So when someone says two raters agree 95% of the time, ask for three things: the marginals, the kappa, and the failure cell.

Turning disagreements into boundary rules (guideline v1 to v2)

The disagreements from Rule 3 are not noise to average away. They are the most direct map you will ever get of where your definition is vague.

In the first lesson, two people label the same emails blind and split on 20 of them. One reads: "We expect to ship Friday, pending customs." Is that a delay? Nobody was careless. Guideline v1 simply did not say what to do with a conditional date.

When the team sorts those 20 disputes, most land in a few gaps in the rules. The course shows it with an illustrative before and after: labellers disagree on about one email in five in week one, the team adds three boundary rules, and in week two disagreement falls to about one in twenty. Measured accuracy jumps, not because the model changed, but because the answer key stopped contradicting itself.

The meeting is simple to run. For each disputed item, put the email, both labels, and the sentence of guideline v1 each labeller relied on side by side. Group the disputes by situation, not by labeller. For each group, the guideline owner writes one boundary rule with an example on each side of the line. Publish a new version, relabel the affected items, and stamp them with it.

Here is what a v1 to v2 diff can look like for the delay task. The rule wording is illustrative, written to match the situations the lessons name.

Guideline v1
  DELAY: the email indicates the shipment will arrive after the agreed date.

Guideline v2 (adds three boundary rules)
  R1 Conditional dates. "We expect to ship Friday, pending customs" is NOT DELAY
     unless the email also says the agreed date will be missed.
  R2 Several orders in one email. Label DELAY if any order in the email is
     delayed, and list the affected order numbers in the notes column.
  R3 Very short replies with no date ("noted, will update") are NOT DELAY.
     The labeller may not infer a delay from tone alone.

Two habits make this safe. First, the version on every label row lets you find every item labelled under v1 that touches R1, R2 or R3 and re-check exactly those. Second, measure kappa again on fresh items labelled under v2, not on the 20 items everyone just discussed. Agreement on items argued over in the meeting tells you the meeting happened, not that the guideline works for a new labeller.

Time splits: why a shuffled split tests memory

Once the labels are trustworthy, the last question is which items go into which set. The default is to shuffle the pile and cut it in two. Fair, and still a trap.

Think of a practice book with the answers printed in the back. A student who finds them aces the practice and fails the exam. Leakage is the same thing in a dataset: near-copies land on both sides of the split, so the test measures memory instead of skill.

Supplier email is a textbook case. One supplier sends the same template all year. A shuffle puts its emails in both dev and test, so every improvement made on dev looks like it generalises. The fix is to split by whole supplier, or to keep later months for testing, ideally both.

The module asks for three parts: a development set you iterate on as often as you like, a test set drawn from later months, and a holdout kept sealed until the release decision. Each supplier lives on one side only.

This changes what the numbers mean. If a new supplier signs on, a split by supplier tells me something about how the system handles strangers. A shuffled split mostly tells me how it handles familiar templates. A split by time also catches drift: new policies and phrasing arrive over the months, and only a later-months test set sees them before users do. I make the same argument for agents more broadly in agents need an independent exam, and the sealed holdout is the same discipline I describe in the release gate is the product.

The check is mechanical. Every test item is dated later than every dev item. No supplier appears on both sides. Nothing synthetic appears in test. Each is a one-line spreadsheet check that catches a failure no model tuning will reveal.

Run the lab, then go deeper

The lab for this module is blind double-labelling, and you can reproduce it with a spreadsheet and a partner, or with yourself in two passes a day apart.

1. Take a 40-item sample of supplier emails and write guideline v1: a definition of delay, a few examples, and any boundary cases you already know.
2. Label all 40 blind against v1. Neither labeller sees the other's labels or any model output. Record one row per item with these columns: id, supplier, email date, guideline version, labeller A, labeller B.
3. Compute kappa from the four cells of the 2x2 table, using the block below.
4. Sort your disagreements, write three boundary rules, and publish them as guideline v2. The deliverable is the labels file, the kappa result, and the v1 to v2 diff.
5. Check the result: every row carries a guideline version, and if you later split the set, every test item is later in time than every dev item.

The course's Lab B calculator takes a CSV (id, slice, human, judge, control); in labeller mode, labeller A goes in the human column and B in the judge column. A spreadsheet does the same work, with labels (DELAY or NOT DELAY) in columns E and F and rows 2 to 41.

H1  both delay        =COUNTIFS(E2:E41,"DELAY",F2:F41,"DELAY")
H2  A delay, B not    =COUNTIFS(E2:E41,"DELAY",F2:F41,"NOT DELAY")
H3  A not, B delay    =COUNTIFS(E2:E41,"NOT DELAY",F2:F41,"DELAY")
H4  both not          =COUNTIFS(E2:E41,"NOT DELAY",F2:F41,"NOT DELAY")
H5  N                 =SUM(H1:H4)
H6  observed          =(H1+H4)/H5
H7  chance            =((H1+H2)/H5)*((H1+H3)/H5)+((H3+H4)/H5)*((H2+H4)/H5)
H8  kappa             =(H6-H7)/(1-H7)

Before trusting your sheet, feed it cells with known answers. The slide table (80, 5, 3, 12) must give 0.70. The paradox table (940, 20, 30, 10) must give 0.26. The lab's first sample, B1, has 20 scored items with cells 12, 1, 3, 4 (12 both pass, 1 passed by people only, 3 passed by the judge only, 4 both fail). Observed agreement is 16 / 20 = 0.80, chance is (13 x 15 + 7 x 5) / 400 = 230 / 400 = 0.575, and kappa is 0.225 / 0.425 = 0.53.

Then run a negative control, the same one the lab's test suite uses. Replace the chance formula with a fixed 0.5, as if the labels were balanced. On B1 that gives (0.80 minus 0.50) / 0.50 = 0.60 instead of 0.53. If your sheet cannot tell those apart, it is not computing kappa.

B1 also shows why Rule 4 asks for per-group results. Split by language, the 12 English items give a kappa of 0.80 and the 8 German items give 0.25. Both slices are under 30 items, so read them with care, but an overall 0.53 hid a strong slice and a weak one.

Both lessons, "An answer key you can trust" (v04) on building the gold set and v05 on reading kappa, belong to the module Gold sets and labels, and you can watch them at https://learnai.ericmacdougall.com/engineering-videos/. The module closes on the rule I now apply before I believe any accuracy figure: no accuracy claim counts until the gold set behind it is blind, double labelled, versioned, stratified and split by time and source. Measure agreement with kappa before you blame the model, and back the gold set with outcome labels wherever you can.

Go deeper: the full lesson and lab are in the Transition2 course.