What a confidence score should mean
When an AI analyst says “Harmless, 90% confidence”, it is making two claims: what it thinks happened, and how often it expects to be right when it says that with that much certainty. The second claim is the one most teams never check, and it is the one that decides whether the number is worth showing at all.
A confidence score is useful when it is honest. If verdicts given at 90% turn out right about nine times in ten, a reviewer can plan around that: spend less time on the sure ones, more on the unsure ones, and know roughly how many mistakes are hiding in each pile. If the same 90% turns out right only six times in ten, the number is worse than useless, because it invites people to relax exactly where they should not.
Confidence is also different from severity. A verdict can be “Real attack, 55%” on a server that runs payments, and that alert deserves more attention than “Harmless, 95%” on a test laptop. The score tells you how sure the agent is, not how much is at stake.
It helps to separate two kinds of doubt. One is doubt about the meaning of the evidence: the records are all there, and they could be read two ways. The other is doubt about whether the evidence is complete: a log source was missing, or a search timed out. A good verdict says which kind of doubt it carries, because the fixes differ. The first calls for a person’s judgement; the second calls for someone to go and get the missing data.
Calibration: does 90% mean nine in ten?
Calibration is the match between stated confidence and how often the verdict is right. You measure it by grouping verdicts into bands by confidence, such as 50–69%, 70–89% and 90–100%, and for each band counting how many verdicts a reviewer later agreed with. In a well-calibrated system, the share that held up in each band sits close to the band’s stated confidence.
An illustrative example, with made-up numbers rather than results: over a month, an agent gives 200 verdicts at 90–100% and reviewers change 8 of them, so 96% held up. That band looks honest. In the 70–89% band it gives 100 verdicts and reviewers change 35, so only 65% held up. That band is overconfident, and reviewers should treat a “75%” as a coin toss with a slight lean until the system improves.
Miscalibration comes in two kinds. Overconfidence, where stated numbers are higher than reality, is the dangerous one, because it hides mistakes. Underconfidence, where the agent says 60% about things it nearly always gets right, wastes reviewer time but rarely causes harm. Language models left to themselves tend towards overconfidence and towards round numbers, so a score that has never been checked is best read as a ranking, not a probability.
A ranking is still useful. Sorting the queue so that the least certain verdicts get the most attention works even before the numbers are trustworthy as probabilities. What it does not support is a rule such as “close anything above 90% without review”, which only makes sense once you know what 90% means on your own alerts.
Where the number comes from
There are a few ways to produce a confidence score, and it is fair to ask any vendor which one they use. The simplest is to ask the model how sure it is. That is cheap, and the number often tracks the strength of the evidence, but it inherits the model’s habits: favourite values, and a lean towards sounding sure.
Better methods tie the score to things that can be checked. Did the agent find records that directly support the verdict, or is it inferring from absence? Did a second, independent read of the same evidence reach the same verdict? Were key data sources missing? A verdict reached on direct evidence from several sources deserves more confidence than one reached because a search came back empty.
Whatever the method, the score should be explained. “90% because the process chain leads to a known adversary-simulation agent and the network events show a regular five-minute beacon” is a number a person can weigh. “90%” on its own is a number a person can only accept or ignore.
How reviewers should use it
Confidence is a tool for spending review time well. It should change how deeply a person reads a verdict, not whether anyone reads it.
One habit is worth naming. Low confidence is not a failure of the agent. A verdict marked unsure, with a clear account of what data was missing, is the agent doing its job honestly. The failure to watch for is the opposite: high confidence with thin evidence.
Resist the pull towards closing high-confidence harmless verdicts without reading them. It is tempting once the numbers have looked good for a few weeks, and it is exactly how an overconfident band does its damage. If volumes make full review impossible, decide openly which verdicts get a light check and which get a full one, write that down, and keep sampling the light-check pile.
- Read real-attack verdicts first, whatever their confidence. A 60% real attack is still the most urgent thing in the queue.
- For high-confidence harmless verdicts, read the reasons and check the one record they rest on. This is quick when evidence is cited.
- For middling confidence, read the whole investigation and look at which questions were not asked.
- For low confidence, treat the verdict as a request for help. The agent is telling you what it could not settle.
- Weigh confidence against what is at risk. A harmless verdict on a crown-jewel system deserves a longer look than the same score on a test machine.
- When you change a verdict, say why. The reason is what makes the correction useful later.
Checking calibration with your own corrections
You do not need a data science team to check calibration. You need each verdict, its confidence and the reviewer’s final call, kept together. From there the method is a spreadsheet exercise.
Two warnings. First, the check is only as good as the reviews. If reviewers agree with everything to clear the queue, calibration will look perfect and mean nothing. A small blind sample, where a reviewer judges the alert before seeing the agent’s verdict, guards against that. Second, small numbers mislead. Ten verdicts in a band tell you very little; wait for dozens before drawing conclusions, and look at the trend over months rather than any one week.
- Export a month of verdicts with confidence and the reviewer’s decision.
- Group them into confidence bands and count how many held up in each.
- Compare each band’s hit rate with its stated confidence.
- Split by verdict type and alert type. An agent can be well calibrated on harmless verdicts and overconfident on authorised activity.
- Repeat after any change: a new model version, a new data source, or a new kind of alert.
- Feed what you find back: tell reviewers which bands to trust less, and tell your vendor.
Where armor360 fits
armor360 gives each verdict (Real attack, Suspicious, Harmless, authorised activity or unsure) with a confidence percentage and the reason, and cites the records behind it. A person agrees with the verdict or changes it, and that review is recorded, so verdicts, confidence and reviewer decisions can be compared over time.
In our lab investigation of a simulated attacker on linux04, the verdict was Real attack at 90% confidence, with twenty-two records kept as evidence.
