Reaction Time Test

"Best of five" is not your reaction time

Reaction times are lopsided. There is a hard floor — nerve conduction and muscle activation take what they take — and a long tail of moments where your attention drifted. Taking the fastest of several attempts does not average that noise away. It deliberately picks the luckiest sample, so it is biased fast by construction, and the bias grows the more times you try.

Simulating a realistically skewed distribution with a median of 236 ms and running many rounds of each attempt count:

Best-of-five reads 37 ms fast. Best-of-twenty reads 55 ms fast. The median of exactly the same attempts is within 2.0 ms of the truth at every count — which is the control that makes the first column trustworthy rather than an artefact of the simulation.

This is why reaction-time leaderboards mean very little. Everyone posts a personal best, the bias grows with persistence, and the ranking ends up measuring how many attempts somebody was willing to make. If you want a number that describes you, take the median of about ten and throw the fastest one away.

So why does the board above rank the mean, not the median?

That last sentence is advice about describing yourself, and it does not settle what a board should rank, because a board has a second problem the panel above ignores: nobody has to accept their first run. Anyone can simply go again. What matters for a ranking is therefore not only where a statistic sits, but how far one person's posted number drifts when they repeat the run — and the honest way to compare those is on the same attempts rather than on two separate samples. Every row below is measured on an identical stream of tries.

The first row is the panel above restated: best-of-five sits 38 ms below the truth before anybody has repeated anything. The mean of five is within 5.4 ms, so it is not carrying that bias.

The surprise is the third row. Switching the board to the median of five would make it worse: at five tries the sample median is the wider statistic, not the steadier one — 73 ms between a lucky run and an unlucky one against 65 ms for the mean. A median throws away everything except the middle value, and with only five to choose from that costs more information than the tail resistance is worth. Being wider, it also hands more to whoever repeats the run: 35.6 ms against 32.5 ms over 25 attempts. The intuition that a median is always the robust choice is simply not true at this sample size.

What does help is more tries. Ten of them cuts the spread from 65 ms to 46 ms and cuts what repeating buys by 8.8 ms, which is why the ten-try run is offered above and keeps its own board. It is the same task done for longer rather than a different one, and it is the sterner of the two. None of this makes either board immune — 25 runs still buys 24 ms at ten tries, and no summary of a noisy quantity ever fixes that completely.

Each row is 20,000 simulated runs for the spread, and 800 simulated players repeating the run 25 times for the last two columns, drawn from the same skewed distribution as the panel above. The seed is fixed so the table is identical on every build; the comparison was checked across seven seeds first, and the median of five came out wider on all of them.

And part of your score belongs to your screen

The stimulus cannot appear until the display draws its next frame, so some of the delay is over before you have seen anything at all. Then your click waits for the next input poll. Neither of those is you.

Add the input side — a standard 125 Hz USB device averages another 4 ms — and the totals come out at about 12.3 ms on an ordinary 60 Hz laptop against 2.6 ms on a 240 Hz screen with a 1000 Hz mouse.

The same person, reacting identically, scores about 10 ms better on the second setup. That is roughly a fifth of the interquartile spread of the scores themselves — not a rounding error, and more than enough to reorder a leaderboard.

Two useful consequences. Comparing your score to a stranger's is close to meaningless unless you know their hardware, but comparing your score to your own last month is fine, because the latency is a constant offset that cancels. And if you want to know your actual reaction time rather than your score, subtract about ten milliseconds on ordinary equipment — the number you see was never entirely yours.