Understanding Quantitative Confidence Scores

Every model output should come with a statement of how much to trust it. The interesting question is what that statement is actually derived from.

Confidence is about the estimate, not the outcome

A common confusion: a confidence score is not the probability of the event. A model can be highly confident in a 30% probability. Those are two separate quantities.

The probability describes the outcome. The confidence describes how much to trust the probability — the model's own assessment of the reliability of its inputs and assumptions in the current environment.

Estimator agreement as a defensible basis

One of the more honest ways to build a confidence score is to compute the same quantity several independent ways and measure how much the estimates agree.

For volatility, you might estimate it from 20-day realized returns, from 90-day realized returns, and from an average true range calculation. When all three land close together, the volatility regime is stable and any of them is a reasonable input — high confidence. When they diverge sharply, the regime is in transition and no single number deserves much weight — low confidence.

The appeal of this approach is that the score is derived from measured data rather than asserted. It is reproducible, and it fails visibly rather than silently.

Other legitimate sources of a confidence signal

Sample size. An empirical frequency computed from 500 observations deserves more weight than one from 12. Any score should degrade when the underlying sample is thin.

Regime similarity. If current conditions resemble the data the model was calibrated on, confidence should be higher than if conditions are unprecedented in the training window.

Model-versus-empirical agreement. When a theoretical model and a purely historical frequency count produce similar answers, the estimate is standing on two legs rather than one.

How to spot a decorative score

Ask what would make the number go down. A genuine confidence score has a clearly stated mechanism that can be adversarially checked — feed it conflicting inputs and it should fall.

Warning signs include a score that is always high, one with no documented derivation, one that never contradicts the headline output, and one presented with implausible precision. Confidence expressed as three coarse bands is usually more honest than a score reported to two decimal places, because the underlying uncertainty about uncertainty is itself large.