When the Sample Is Tiny, Show No Number
‘100% of the one fan who tried got it right’ is true and tells you nothing. When the sample is tiny, I decided to show a blank instead of a number.
Statistics make a screen feel full. Add "here is how other fans answered" after someone picks an answer, and a question they solved alone becomes one they solved alongside others. So the urge to add numbers is always there. The catch is that the moment a number appears, it starts making a claim.
In DUGOUT 9, an independent fan project for KBO, the Korean professional baseball league, I stopped myself before adding two numbers: the answer rate on the daily quiz, and the win rate in a fan's record of games attended. Both were calculated correctly. At certain moments, both correct numbers were lying.
When a true sentence misleads
DUGOUT 9 has a quiz that is rebuilt every day, and when you check your answer it shows how other fans did: the share who got it right and the most common wrong answer. With enough answers, that is a good feature. Knowing that others missed the question you missed takes the sting out, and getting right what most people got wrong feels good.
But early in the morning when a question has just opened, or while hardly anyone knows the product exists, one or two people have answered. Then the screen says "100% of the 1 fan who tried this got it right." The arithmetic holds. The reader, though, takes it to mean "this question is easy". The range in which the sentence is true and the range in which it is read are nowhere near each other.
The same thing happened in the attendance record, where a fan logs the games they went to see at the ballpark and gets a personal win rate. If none of those games has finished, or none has a result yet, the formula returns 0%. And 0% does not read as "no data yet". It reads as "when this fan goes, the team loses". It was attaching a number that looks like a fact about the person, with nothing behind it.
What makes this hard to catch is that nothing is wrong in the usual sense. The formula is right, the tests pass, and the number on screen matches the data exactly. The gap is not between the data and the number. It is between the number and what a person concludes from it, and no test of the calculation will ever reach that gap.
Accepting the blank
So I chose to show nothing in both places. The answer distribution is not drawn below a sample of ten. The attendance win rate shows "—" rather than 0 when no game has been decided.
That costs something. Early screens look empty. The distribution is missing exactly when the social context it provides would matter most, and a fan who has just started a record sees a dash where a number would be. Choosing the fuller-looking screen would have been far easier.
I still judged the blank better, because the two options damage different amounts. A blank states "not yet", accurately. A 100% from one person and a 0% from no games give a wrong impression that looks precise. Someone who has been misled once doubts the number even after the sample is large enough to mean something. A number made up to fill an empty state is not information. It is an optical illusion.
Decide when the sentence becomes true
The order I took from this: before adding a statistic, decide from what sample size the sentence it produces is true, and below that, draw no number at all.
- For any rate, decide the denominator at which it starts to mean something.
- Do not display what the formula returns when the denominator is zero. Zero and "none" are different values.
- Design what the screen shows below the threshold — a blank, a dash, nothing — at the same time.
There is nothing special about ten. What matters is that a threshold exists. Without one, a number appears the moment it can be calculated, and that moment is usually when it means the least. Set the threshold later and you also pay to win back trust in the numbers that were misread in the meantime.
An empty screen can be honest
It is easy to treat an empty screen as a failure. I did. What changed my mind this time is that a blank does not lie. It says "not yet", and that is always true. A dash in a fan's own record does not tell them anything about themselves; a zero did, and it was wrong.
The numbers will come as answers accumulate. For people to trust them when they do, the numbers shown before them must not have spent that trust in advance. Showing nothing when there is nothing worth showing is also a way of handling data.