Counts out of a thousand: what a flag really means

Here is a sentence you will hear in a detector demo: "it catches 90% of them." A record gets flagged this morning, and the natural reading is that the record is 90% likely to be a real problem. Hold it, move on.

That reading is wrong more often than it is right, and nothing about the demo was false. The sentence answered a different question from the one you asked.

The habit that fixes this needs no formulas. Whenever someone gives you a percentage, turn it into things: people, records, emails, runs. Count them out of a thousand at the real base rate. Then decide what the flag is allowed to do.

This essay walks through the counts from module M02, "Counts out of a thousand": a disease test, a duplicate-record flagger and a five-step agent chain. The appendix lays it all out as a spreadsheet.

90% of what? Four numbers hiding in one

The lesson starts at a lake. A hundred trout swim among 900 other fish. You want trout, so you lower a net and lift it out with 120 fish in it, 80 of them trout.

Two questions matter, and they point at different places.

Look in the catch jar first. Of what you pulled up, how much is trout? That is precision: 80 of 120, or 67%. Of what we flagged, how much was right.

Now look back at the lake. Of all the trout in there, how many did you catch? That is recall: 80 of 100, or 80%. Of what mattered, how much did we catch.

Lay the whole lake out as counts and you get four numbers, not one:

The 40 comes from 120 minus 80, the 20 from 100 minus 80, and the 860 from 900 minus 40. Every rate anyone quotes about this net is a ratio of two cells in that table. Recall is 80 over the trout column. Precision is 80 over the net row. A combined score like F1 (2PR divided by the sum of P and R) comes out at about 73%, which is tidy but still hides which of the two moved.

A bigger, finer net raises recall and drags up every boot and minnow with it. Drain the lake and recall is perfect while precision is a mess. So when a demo gives you one number, it has not finished its sentence. I want both on the table before agreeing to anything.

The question underneath is conditional probability: a chance that depends on a "given that." Given that a record is a duplicate, how often is it flagged? That is what the demo told you. Given that it is flagged, how often is it a duplicate? That is what you asked. The slide in the lesson puts it in one line: P(positive given sick) is 90%, P(sick given positive) is 9%. The rest of this essay is about how those two can be that far apart.

The disease test: 9 of 98 positives are sick

A disease affects 1 in 100 people. The test catches 90% of the sick and flags 9% of the healthy by mistake. You test positive. Most people guess they are 90% likely to be sick.

Count instead. Take 1,000 people:

• 10 are sick (1,000 x 1%), and 9 of them test positive (10 x 90%).
• 990 are healthy, and 990 x 9% is 89.1, so about 89 get a false positive.
• 9 plus 89 is 98 positives, and only 9 of them are sick.

That is about 9%, or roughly one in eleven. Not ninety.

Nothing is wrong with the test. The problem is the size of the two groups. Nine percent of a huge healthy group outnumbers ninety percent of a tiny sick group, because there are so many more healthy people available to be wrongly flagged.

Now move the same test to a clinic where far more of the people walking in are sick, so that 700 of every 1,000 are healthy and 300 are sick:

• 300 x 90% gives 270 true positives.
• 700 x 9% gives 63 false positives.
• 270 of 333 positives are sick, about 81%.

Same test, same 90% and 9%. A positive result meant about 9% in the general population and about 81% in the clinic. The base rate, how common the thing was before you looked, decided what the flag meant.

This is the pattern for anything rare: fraud, duplicate supplier records, dangerous content. A detector that sounds excellent can produce mostly false alarms while every number in its spec sheet stays true.

The duplicate flagger: 190 of 1,188 flags are real

The lesson's second worked example moves this into a system engineers actually build. A records system flags suspected duplicate supplier records. The numbers are illustrative:

• Two records in a thousand really are duplicates.
• The flagger catches 95% of them.
• It wrongly flags 1% of normal records.

Run it over 100,000 records:

The arithmetic: 100,000 x 0.2% is 200 duplicates, and 200 x 95% is 190 caught. That leaves 99,800 normal records, and 99,800 x 1% is 998 false flags. 190 plus 998 is 1,188 flags, and 190 divided by 1,188 is about 16%. Roughly one flag in six is real.

The narration in the lake-and-net lesson tells the same story with rounder numbers: 20 duplicates in a thousand, a detector that catches most of them and wrongly flags about one in ten good records. It lands in the same place, about one flag in six. Then it moves the identical detector to a messy batch where one record in five is a duplicate. With 200 duplicates in 1,000, catching 90% gives 180 real flags, and one in ten of the 800 good records gives 80 false ones. 180 of 260 is about two flags in three. You would want to check that the detector really behaves the same on the messy batch, but the point stands: same detector, different base rate, different meaning.

There is one more trap, and it is the one that hides best in dashboards: overall accuracy. Accuracy is the share of all decisions that were correct. Imagine a lazy flagger that never flags anything. On the 100,000 records it misses all 200 duplicates and is right on the other 99,800, which is 99.8% accuracy. The real flagger is right on 190 plus 98,802, which is 98,992, or about 99.0%. The lazy one wins on accuracy while catching nothing.

When the thing you care about is rare, doing nothing looks excellent. So split the results by case type and by how much a miss would hurt; a single accuracy figure treats every miss as identical.

Compounding steps: 0.95 to the fifth

The same counting habit applies when checks happen one after another instead of side by side. An agent run that has to get five steps right, each at 95%, does not succeed 95% of the time. Multiply.

Start with 100 attempts:

In one line: 0.95 to the fifth power is 0.774, so about 77 of 100 runs survive. Each step sounds reliable. The chain loses almost a quarter of its runs.

It gets worse with length. Ten steps at 95% gives 0.95 to the tenth, 0.599, so about 60 of 100. A twenty-step desktop task at 98% per step finishes about 67% of the time. Push each step to 99.5% and twenty steps finish about 90% of the time, while at 95% per step the same twenty steps finish only about 36% of the time.

Two lessons follow. A per-step number is not a task number. And long chains need checks along the way, so a failure at step two gets caught at step two instead of surfacing as a confusing result at step five. I wrote more about keeping the probabilistic parts of a system apart from the parts that must behave the same way every time in separate probabilistic development from deterministic shipping.

Multiplication carries an assumption, and the lesson calls it shared batteries. Two smoke detectors that each miss one fire in ten miss the same fire only one time in a hundred (0.1 x 0.1), but only if they fail for unrelated reasons. Give them a shared dead battery and the second detector adds almost nothing. Two copies of the same model checking each other's work are a shared battery. When you stack checks to get safety, the multiplication is only as good as the independence.

Block, note or route: deciding from precision at the real base rate

Back to the flagger, because this is where the numbers become an engineering decision. The same detector can sit behind three very different actions.

Block. The flag stops the record automatically. At one flag in six real, five blocks in six land on honest suppliers: 998 of the 1,188 blocks. That means stalled work and angry calls, and one in six is nowhere near good enough.

Note. The flag adds a note for a clerk who is already looking at the record. A false alarm takes a few seconds of attention, and one in six real is perfectly fine.

Route. The flag sends the record to a review queue. Now the question is review time: 1,188 flags per 100,000 records, of which 998 are false, all landing on the same people. Whether that works depends on the size of the team and how often the queue fills.

Same detector, three actions, three different verdicts. So ask what happens next before you call a number good.

The course makes the trade explicit with an illustrative weighting in harm units. Suppose a missed duplicate does 100 units of harm and a false alarm does 1 unit. A relaxed threshold that misses 9 and raises 100 false alarms scores 9 x 100 plus 100 x 1, which is 1,000 units. A strict threshold that misses only 2 but raises 300 false alarms scores 2 x 100 plus 300 x 1, which is 500 units. The strict setting wins here, but only because a false alarm that becomes a note takes a few seconds of attention. Move the same flags into automatic blocks, raise the weight of a false alarm, and the answer can flip. Write the weights down before you pick the threshold, not after.

Three cautions apply to the precision figure itself.

First, measure it at your base rate, on your data. The vendor's precision came from the vendor's mix. Your 2 in 1,000 or 1 in 5 is what decides the meaning of a flag in your system.

Second, small samples swing. A town has a hospital with 45 births a day and another with 15. The small one sees days with at least 60% boys about 30% of the time; the big one about 12%. A perfect score on 15 examples tells you much less than it looks. Even 20 correct flags out of 20 only supports a lower bound of about 84% precision (20 divided by the sum of 20 and 3.84).

Third, rare harms hide from inspection. Say an agent does something seriously wrong once in 2,000 conversations, and the team reviews 50 a week. The chance the bad one shows up in a given week's sample is 1 minus (1,999/2,000) to the 50th, about 2.5%. Ninety-seven weeks out of a hundred, review says all clear. For rare, high-harm failures, the control has to be a limit built into the system that prevents the harm, not an inspection that might spot it.

If the detector reports a confidence score, the calibration lesson adds the last piece. A forecaster's 70% chance of rain is a claim about many days: across a hundred days she says 70%, it should rain on about seventy. You audit a model's 90% the same way: gather a fair sample scored around 90, have people check it, and repeat for 80 and 70. Then check the slices that matter, such as supplier and document type, because a bucket that holds up overall can hide one supplier quietly failing at 70. Write a threshold per slice and auto-pause automation for any slice that drops below it, sending that slice to people until the numbers recover. I went through what it took to decide whether a detector actually works on a test set we wrote in we measured local safety models; the hard part was deciding what works.

The rule I take from the module fits in a sentence: turn every rate into counts at the real base rate before you decide what a flag is allowed to do. Ask ninety percent of what, count it out of a thousand, and let the base rate decide whether a flag blocks, routes or just leaves a note.

Worksheet appendix: reproduce the lab in a spreadsheet

The lab is a counts worksheet: five percentages to convert, the duplicate flagger and a five-step agent chain, finished with a one-line action decision for the flagger. It takes about 25 minutes. Work in counts, not percentages, and check yourself against the answer key: 9 of 98 positives sick, 190 of 1,188 flags real, and about 77 of 100 five-step runs intact.

Part 1, convert rates to counts. Put a population in one cell and each rate beside it, then multiply. Five rates from the lesson to practice on, each against a population of 1,000:

Part 2, the disease test. In a sheet, set A1 to 1000, A2 (base rate) to 0.01, A3 (catch rate) to 0.9, A4 (false alarm rate) to 0.09. Then:

• A5, sick: =A1*A2 gives 10
• A6, true positives: =A5*A3 gives 9
• A7, healthy: =A1*(1-A2) gives 990
• A8, false positives: =A7*A4 gives 89.1
• A9, share of positives sick: =A6/(A6+A8) gives about 0.09

Change A2 to 0.3 and A9 becomes 270 over 333, about 0.81. That one edit is the whole lesson.

Part 3, the duplicate flagger. Same sheet layout with A1 at 100000, A2 at 0.002, A3 at 0.95, A4 at 0.01. You should see 200 duplicates, 190 caught, 99,800 normal records, 998 false flags, and 190 of 1,188, about 0.16. Add a row for lazy accuracy, =(A1*(1-A2))/A1, which gives 0.998, and one for the flagger's accuracy, =(A6+(A7-A8))/A1, which gives about 0.99.

Part 4, the five-step chain. Put 100 in B1 and =B1*0.95 in B2, then fill down to B6. You should read 95, 90.25, 85.7, 81.5 and 77.4. Check against =100*0.95^5. Change the exponent to 10 and you get about 60.

Part 5, the decision. Write one line saying what the flagger should do, from its precision at the real base rate. Mine: at about one flag in six real, the flagger adds a note for the clerk and never blocks a supplier on its own.

Go deeper

The two lessons behind this essay, "Count before you believe" (v02, the lake and the net) and "Confidence you can check" (v07, the forecaster and the router) from module M02, "Counts out of a thousand", are in the engineering video library.

Go deeper: the full lesson and lab are in the Transition2 course.