A Rate Is a Belief With a Width: Updating From Counts to Live Traffic

Two shooters at the foul line both hit 80%. One made 4 of 5. The other made 80 of 100. The scoreboard calls it a tie, and nobody watching believes that. What we actually believe about the two shooters is different, and the difference is not in the center of the number. It is in the width.

That is the whole thesis of this essay. A rate is a belief with a width. Pseudo counts make that belief explicit, pooling rescues thin slices, and probability matching routes traffic while you learn, as long as you set a stopping rule first.

None of this needs special software. Every example below is small enough to rebuild in a spreadsheet, and I give every count so you can check my arithmetic.

A forecaster, a test, and two very different positives

Start with a weather forecaster in late October. At that time of year it rains about three days in ten, so 30% is where the forecaster starts. Clouds roll in, and the number goes up. The radar shows the storm turning away, and it comes back down. The forecaster never jumps to certainty and never ignores the sky.

That is all belief updating is. You hold a belief before the evidence, called the prior. You look at what arrived. You move by exactly as much as that evidence deserves, and where you land is the posterior. The arithmetic that connects the two is a single rule of conditional probability, and it is far easier to follow in counts than in symbols.

Take 1,000 people where 1% are sick: 10 sick, 990 healthy. A test catches 9 of the 10 sick people. It also flags 9% of healthy people by mistake, and 990 x 0.09 = 89.1, so call it 89 false positives. That makes 9 + 89 = 98 positives in total, and only 9 of them are sick. A positive result means about a 9% chance of being sick.

Now carry the same test into a specialist clinic where three in ten patients are sick. Out of 1,000 patients, 300 are sick and the test catches 270 of them. The 700 healthy patients produce 700 x 0.09 = 63 false positives. A positive now means 270 of 333, about 81%.

Same test, same chemistry. Only the prior changed. The chance of the thing given the evidence is the cases with both, divided by all the cases with the evidence, and "all the cases with the evidence" depends on what walks through the door.

This is the first habit I want engineers to build. When a classifier raises a flag in production, ask what the incoming traffic looks like before you decide what the flag means.

There is a second view of the same update, using odds. A positive is ten times more likely in a sick person than a healthy one (90% against 9%). So a positive multiplies the odds by ten. In the general population the odds start near 1 in 100 and move to about 1 in 10, which is the 9 to 89 we just counted.

Updates also chain. Today's posterior is tomorrow's prior. A second independent positive multiplies the odds by ten again: 9 to 89 becomes 90 to 89, about 50%. A third independent positive takes it above 90%. How far each step moves you depends on the strength of the evidence and on where you started.

The forecaster has one more lesson. A forecast of 70% is not wrong on a dry day. Gather every day that got a 70, say a hundred of them, and if the number means what it claims, about seventy see rain and thirty stay dry. That property is calibration. It is necessary, not sufficient: a forecaster who says 30% every single morning in a town where it rains three days in ten is perfectly calibrated and tells you nothing about which day needs an umbrella. You want calibration and sharpness together, where sharpness means saying 10% on Tuesday and 90% on Thursday when the evidence supports it, with both numbers holding up when you count.

Pretend shots: writing a belief down as counts

Back to the foul line. You could keep the 80% and add a note that says "small sample". The trouble is that a note cannot change a number, and someone eventually has to act on a number.

The trick is to pretend, before any real shots, that you have already watched some. Say 7 makes and 3 misses, because that is roughly how you think a typical shooter performs. That is the prior, written as counts. Then add the real shots straight on top.

The first shooter's 4 makes and 1 miss bring the tally to 11 makes and 4 misses: 11/15, about 73%. The second shooter's 80 makes and 20 misses become 87 and 23: 87/110, about 79%. They are no longer tied, and the reason is visible in the counts. Ten pretend shots against five real ones means the pretending outweighs the watching. Ten pretend shots against a hundred real ones barely register.

This recipe, pretend rights and wrongs plus real rights and wrongs, describes a belief about a rate called a Beta distribution. "Distribution" only means how your belief spreads across the possible rates. In a business setting the prior is where assumptions stop hiding. Someone has to write down what they expected before the results came in, and everyone can argue with it.

The center is not enough to decide on, because the belief also has a width. Take the range holding 95% of the updated belief. For the first shooter it runs from about a coin flip to above 90%. For the second it is far tighter. That range is a credible interval, and the sentence that goes with it is precise: given this prior and this model, 95% of the belief sits inside. A confidence interval is a different object (it describes how a procedure behaves across many repeats), so keep the two words apart. For comparison, the classical score interval for 4 of 5 runs from 38% to 96%, and for 80 of 100 from 71% to 87%.

The obvious objection is that I picked the pretend shots. You could pick different ones and get a different answer. That is why you state them and then check. Swap in a gloomy prior of 3 makes and 7 misses. The first shooter becomes 7/15, under 50%. The second becomes 83/110, still in the mid seventies. That is a sensitivity check, and it says something honest: with five shots, the assumption is doing most of the work.

One naming trap catches people in meetings. This Beta has nothing to do with the beta in the F-beta score on a quality dashboard. There, beta is a dial for weighting precision against recall. Here, Beta is a belief about a rate.

Worked example: 49 of 50, and which prompt is better

This is the first lab, the pseudo count worksheet, and it has three cases.

Case 1. A classifier gets 49 of 50 right.
  Gentle prior: 1 right, 1 wrong.  Belief: Beta(50, 2).
  Best guess   = 50 / 52                 = about 96%
  Low end      = BETA.INV(0.025, 50, 2)  = about 89.5%
  High end     = BETA.INV(0.975, 50, 2)  = about 99.5%
  Classical score interval on 49/50: about 89.5% to 99.6%, centre about 94.6%

Case 2. Prompt A: 45 of 50.  Prompt B: 48 of 50.
  Beliefs with the gentle prior: A = Beta(46, 6), B = Beta(49, 3).
  Column A: =BETA.INV(RAND(), 46, 6)
  Column B: =BETA.INV(RAND(), 49, 3)
  Column C: =IF(B2 > A2, 1, 0)
  Fill a hundred thousand rows, average column C: about 87%

The gentle prior of 1 right and 1 wrong says almost nothing beyond "both outcomes can happen". With that prior and decent data, the credible range and the classical interval agree almost exactly. What changes is the sentence you are allowed to say: there is a 95% chance the true accuracy is in this range. That is the sentence most people already assume a confidence interval licenses.

The second case answers the question people actually ask about two prompts. Draw a plausible rate from each belief, see which is higher, repeat many times, and count. B beats A in about 87% of draws. That is a decision number, not a gate. If switching to B is quick to reverse and low in harm, 87% may be enough. If switching takes real effort or is hard to undo, gather more data before you move.

When the prior drowns the data, report both

Priors cut both ways. Say a supplier tells you their model is 95% accurate, and you encode that claim as 95 right and 5 wrong. That is a prior counted as 100 items. You then test 40 of your own emails and get 30 right, which is 75%.

The posterior is (95 + 30) / (100 + 40) = 125/140, about 89%. Your own data says 75%. A prior counted as 100 items outweighs 40 real ones by more than two to one, so the posterior sits much nearer the claim than the evidence. The 89% looks like a measurement and is mostly a restatement of what the supplier said. (This one is an illustrative case from the course, not a real supplier.)

A strong prior is a claim about the world. Use one only when real past data justifies it, and a headline figure about someone else's traffic is not real past data about yours. When you are unsure, start gentle and let your own counts speak.

The defense is a reporting rule that I hold for every posterior I put in front of a reviewer: show what the data alone says, right beside it. The lab's deliverable is the worksheet plus a two-line report built that way.

A two-line report for the prompts reads like this. "Prompt B is better than prompt A with probability about 87%, using a gentle prior of one right and one wrong. Data alone: A 45 of 50 (90%), B 48 of 50 (96%)." Anyone reading it can see both the belief and the evidence it rests on, and can challenge the prior separately from the data.

The course also draws a practical line about which kind of interval to use. For gates and audits that other people must accept without debate, use classical intervals. For decisions, ranking and monitoring, credible ranges and probability of better are more useful. With plentiful data the two should agree. If they disagree strongly, look at the prior first. I wrote about why the gate deserves that extra rigor in The release gate is the product.

Pooling thin slices toward the group

Priors help most when data is scarce and you have honest history. The classic demonstration comes from baseball. A well known study took 18 players, looked at each one's first 45 at-bats of a season, and pulled every early batting average partway toward the group average.

One hitter started at .400. The pulled estimate said about .290. The season ended near .346. For that one hitter, the pulled guess missed by about .056 and the raw one by about .054, so roughly a draw. Across all the players, though, the pulled estimates had about a third of the squared error of the raw ones. A hot start is partly skill and partly luck, 45 at-bats is a small sample, and the group average carries real information about what is typical.

That is partial pooling. Each slice keeps its own record, but thin data gets pulled toward the group more, and thick data barely moves. The same pretend shot arithmetic does the work: the group supplies the prior.

Here is the production version from the second lab. An email router audits its decisions by supplier and by language. Across suppliers, the system runs at about 92%. Treat that 92% as a prior counted as 20 emails, which means 0.92 x 20 = 18.4 pretend correct and 1.6 pretend wrong.

The new supplier is not at 100%. Three for three tells you almost nothing on its own, and starting from what the system usually does is wiser than starting from nothing. When that supplier has 300 audited emails, its own data dominates and the 20 pretend emails stop mattering.

The language slice is the more useful case, because it is the one that triggers bad decisions. On a Monday the dashboard shows that slice at 0%: two emails audited, both wrong. Someone wants automation switched off for that language today. Pooled, the slice sits at about 84%. That says watch it and audit more this week, not panic. If the next 20 go badly, the estimate falls fast and then you act. Pooling protects you from celebrating a lucky streak and from panicking over an unlucky one.

To build the worksheet, make one row per slice with columns for audited and correct, a raw rate of correct divided by audited, and a pooled rate of (correct + 18.4) divided by (audited + 20). Then rerun it with a weaker and a stronger prior strength to see how much that choice moves the thin slices. That is the same sensitivity check as the gloomy shooter prior.

Three cautions keep this honest. First, the strength of 20 is a choice. The group average tells you the center of the prior, not how hard to pull toward it. A full hierarchical model learns that strength and its uncertainty from the data; a fixed count is a simplified illustration. Second, pool only groups that belong together. If six regional support queues do truly similar work, pooling gives steadier numbers, but a queue that handles legal mail is not the same kind of work and should not be lumped in. Third, pooling helps across the group, not for every member. A truly exceptional performer gets dragged toward average too, as the .400 hitter shows.

Pooling pairs naturally with the per-slice monitoring from the calibration lesson. A healthy average can hide one slice that quietly fails, so you audit a sample slice by slice, write down a threshold for each slice before launch, and pause automation for a slice that falls below its line while people handle that traffic. Pooled estimates make those thresholds usable on thin slices, because a slice with two audited emails no longer trips the line on noise alone.

Routing traffic while you learn, and where not to

Pseudo counts also let you choose between options while you are still learning about them. Picture a row of slot machines with unknown win rates and a hundred coins. Lock onto the best-looking machine after three pulls and you may have a dud. Spread the coins evenly and you waste most of them on bad machines. That is the shape of choosing between prompts, models or routes on live traffic.

Probability matching, also called posterior sampling, handles it in four steps. Keep a pseudo count belief for each option. When a request arrives, draw one plausible success rate from each belief. Send the request to the option with the highest draw. Watch the result and update that option's counts. That is the whole method.

The draw table in the lab shows why it behaves well. Prompt A has 8 wins in 10 and prompt B has 1 win in 2. With the gentle prior, A's belief is Beta(9, 3) and B's is Beta(2, 2). Build the same three columns as before: =BETA.INV(RAND(), 9, 3), =BETA.INV(RAND(), 2, 2), and =IF(B2 > A2, 1, 0). The newcomer wins about 18% of draws, so it gets about 18% of traffic.

That 18% is the point. B has so little evidence that its belief is wide, and a wide belief sometimes produces a high draw. The newcomer gets enough traffic to find out whether it is secretly great, without the even split's waste. As evidence builds, beliefs narrow and traffic moves to the better option on its own.

Bandits have a clear home and clear limits. Use them where outcomes arrive fast and a wrong choice does little harm, like deciding which prompt earns more helpful ratings. Do not use them for safety-critical choices, where each wrong routing does real harm and "learning on live traffic" means learning on people. And do not use them where you need a clean fixed comparison for an audit. A bandit's traffic split depends on its own past results, which is exactly what an auditor does not want.

When you do need a planned experiment, the stopping rule is where most teams go wrong. Run an A/B test between two prompts that are actually identical, check every day for ten days, and stop the first time it looks significant. In simulation, about 20% of those tests declare a winner. That is four times the false alarm rate you thought you had (4 x 5% = 20%). The same trap appears with variants instead of looks: test 20 variants and there is about a 64% chance one looks better by luck, since 1 - 0.95^20 = 0.64.

The fix is procedural, not mathematical. Fix the sample size and look once, or agree a stopping rule before the experiment starts. The lab asks for a one-paragraph stopping rule for your next A/B test, written before you start. Mine reads like this: "We compare the current prompt and the candidate on a sample size fixed today, counting the same outcome for both. We look once, when that sample is complete. We switch only if the candidate wins at the threshold written here, and we report the data-only rates for both prompts beside the result. Any early look is recorded and does not end the test."

Weighting reviewers by how often they are right

The last tool turns reliability into weight. Three reviewers label the same item. Two are careful, right 95% and 90% of the time. One is rushed, right 60% of the time. A majority vote treats all three equally, so if the two weaker reviewers agree, they win.

Weight each vote by the log odds of that reviewer being right instead. In a spreadsheet the weight is =LN(p / (1 - p)).

Reviewer at 95%:  ln(0.95 / 0.05) = ln(19)  = about 2.9
Reviewer at 90%:  ln(0.90 / 0.10) = ln(9)   = about 2.2
Reviewer at 60%:  ln(0.60 / 0.40) = ln(1.5) = about 0.4

Best reviewer says "delay"; the other two say "confirmation".
  delay:        2.9
  confirmation: 2.2 + 0.4 = 2.6
  2.9 > 2.6, so delay wins.

Two votes lose to one, and that is correct. The rushed reviewer is barely better than a coin, so the vote carries little information. A reviewer right exactly half the time would get ln(1) = 0, which is the right weight for a coin. The careful reviewer's single vote is worth more than the other two together.

The obvious question is where the reliabilities come from. You rarely have a full gold set. Agreement-based methods can estimate each reviewer's reliability without one, by noticing who tends to agree with the consensus, and seeded known-answer items then sharpen that estimate. I would never let agreement alone stand in for measured reliability when reviewers share the same blind spots, which is the argument in Model consensus is not engineering evidence. Seeded items are what tie the weights to ground truth.

The same idea, weighting by how much you trust a source, applies to evidence over time. Blend new evidence by its noise. Say a quality score has been a steady 90 give or take 2, and one noisy week comes in at 80 give or take 5. Inverse-variance weighting gives the steady estimate a weight of 1/2^2 = 0.25 and the noisy week 1/5^2 = 0.04. The blend is (90 x 0.25 + 80 x 0.04) / (0.25 + 0.04) = (22.5 + 3.2) / 0.29 = 25.7 / 0.29, about 88.6, so about 89. One noisy week moves you a little. Several noisy weeks in the same direction move you a lot, which is the behavior you want from a monitor.

And when each evaluation of a setting, like a threshold or a chunk size, takes an hour of labelled review, the same habit says to stop trying settings by hand. A belief-guided search, often called surrogate-model optimization, keeps a belief about how the setting affects quality and picks the next one to try where it expects to learn most, often finding a good setting in 10 to 20 smart tries.

Put the two modules together and the rules are short. Start from a gentle prior, update in proportion to the evidence, report the belief with its range, and always show what the data alone says. Pool thin slices toward the group, weigh reviewers by reliability, explore only where mistakes do little harm, and never look at an experiment without a stopping rule.

Go deeper

Two lesson videos cover this material: "Beta: a belief about a rate" (v18), on pretend shots, credible intervals and partial pooling (the foul line shooters), and "Confidence you can check" (v07), on calibration, sharpness and drift (the forecaster and the email router). Both are in the engineering video library. The labs take about 30 and 35 minutes in a spreadsheet, and the check values are all in this essay, so you can tell when your worksheet is right.

Go deeper: the full lesson and lab are in the Transition2 course.