AI mistakes are predictable
AI analysts make mistakes, and anyone who tells you otherwise is selling something. The useful news is that the mistakes are not random. They fall into a handful of patterns, each with a cause you can name, and each easier to catch when the agent shows its evidence. A team that knows the patterns reviews faster and catches more than one that simply reads the verdict and hopes.
This post walks through four patterns: missing data read as absence, misread records, overconfidence, and plausible stories that the evidence does not support. Then it sets out review habits that catch them. The examples are illustrative, built from the kinds of alert any SOC sees.
Missing data read as absence
The most common serious mistake is treating “I found nothing” as “nothing happened”. An agent searches for outbound connections from a workstation after a suspicious download, finds none, and concludes the file never called home. But the workstation’s network logs were never collected, or the collector stopped sending three days ago. The search was empty because the data was missing, not because the activity was.
This mistake is dangerous because it produces confident harmless verdicts on exactly the machines you can see least. It is also the one most within a team’s power to prevent, by knowing which machines send which log types and which have gone quiet.
A well-built agent checks before it concludes. Before reading an empty network search as a clean result, it should confirm the host has sent any network events in the window. When it cannot, the honest verdict is unsure, with the gap named.
An illustrative Linux case: an alert fires for a new SSH key added to a root account. The agent searches auditd records for the file change and finds nothing, so it concludes the key was added through an approved configuration tool. In fact auditd on that host had no rule watching the key file. The right verdict was not harmless; it was “unsure, the change was not recorded”, with a recommendation to add the audit rule.
Misreading a record
Security records are full of traps for a reader in a hurry, human or machine. The agent fetched the right record and drew the wrong fact from it.
These mistakes are small in the record and large in the verdict. Confusing the account that requested a logon with the account that was logged on to can turn a password-spraying alert into a story about an administrator’s routine work.
- Time zones. One source logs in UTC, another in local time, and the agent lines up events that were hours apart.
- Subject and target. Windows logon events (4624 and 4625) name both the account that asked and the account being used. Mixing them up blames the wrong person.
- Parent and child. Reading a process ID as the parent when it is the child reverses who started what.
- Similar names. Two hosts called fin-app-01 and fin-app-10, or a user and a service account with almost the same name.
- Truncated or encoded command lines. A long command cut off in the log, or a base64 string the agent did not decode, hides the part that mattered.
Overconfidence and plausible stories
Language models are very good at writing a coherent explanation. That is a strength when the explanation is grounded and a weakness when it is not. An agent that has gathered half the picture can fill the other half with what usually happens, and the result reads just as smoothly as a fully evidenced account.
An illustrative example: an alert fires for a scheduled task created on a file server. The agent finds the task, sees it runs a script from an IT share, and writes “This is likely routine maintenance by the IT team.” Nothing it fetched shows who created the task or whether the script is the one IT wrote. The sentence is plausible. It is also unsupported, and an attacker who copied a script to that share would get the same verdict.
Anchoring makes this worse. Once an agent settles on an early explanation, it tends to ask questions that fit it, rather than questions that could prove it wrong. The verdict ends up confident because the agent never looked for the evidence that would have shaken it.
Human analysts do all of these things too. The difference is volume and tone. An agent makes the same kind of mistake across many alerts, in the same calm voice it uses when it is right, so a reviewer cannot rely on the writing sounding unsure. That is why the checks have to be on the evidence, not on the prose.
Why cited evidence makes mistakes visible
Each of these mistakes leaves a trace when the agent has to cite its records. Missing data shows up as a list of data sources with a gap in it. A misread record shows up when the reviewer opens the cited event and sees a different account name. An unsupported story shows up as a sentence with no record beside it.
Without citations, a reviewer has two choices: trust the verdict, or redo the investigation. Neither is good. With citations, the reviewer can check the one or two records the verdict rests on, which takes a minute, and move on or push back. That is the difference between review as a ritual and review that catches things.
It also makes corrections useful. A reviewer who changes a verdict and says “the agent read the subject account as the target” is giving feedback that can be remembered and acted on. “Wrong” on its own teaches nothing.
Citations help the agent as well as the reviewer. A system that must point to a record for each claim has fewer places to hide a guess, and a second read of the same evidence, by a person or by the agent itself, can test each claim against its record rather than against the story as a whole.
Review habits that catch them
Good review is a short routine, done the same way each time, so that it stays fast and does not depend on mood.
The routine below takes a couple of minutes for a clear verdict and longer only where something looks wrong. That is the right shape: review time should go where the doubt is.
- Find the sentence the verdict rests on and open the record cited for it.
- Check the time window. Were the events the agent joined really close together, in the same time zone?
- Look at which data sources were read, and which ones you would have expected that were not.
- Ask what would make the verdict wrong, and whether the agent asked that question.
- Distrust “likely” and “probably” with no record beside them.
- Give harmless verdicts on important systems a second look, however confident.
- Record a reason with each correction, in a sentence.
- Each week, re-read a small sample of high-confidence harmless verdicts in full.
Where armor360 fits
armor360 keeps each record its agent collects and cites it where it is used, and the investigation map shows which data sources were read and which path backs the verdict. A person agrees with each verdict or changes it with a reason, and those corrections are remembered for later investigations about the same company.
Its coverage view finds machines that have stopped sending logs, with the likely cause, which helps with the missing-data mistake at its source.
