A practical guide to alert triage

What triage is

Triage is the first decision about an alert: does it need more work, how urgently, and from whom. The word comes from medicine, where it means sorting patients by how badly they need care. In a SOC it means sorting alerts by how likely they are to be real and how much harm they could do if they are.

Triage is not a full investigation. Its job is to reach a sound first view quickly, using the information to hand and a few targeted checks. If the answer is clear, the alert can be closed or passed on with a verdict. If it is not, triage says so and sends it to someone who can dig deeper.

Done well, triage is where a SOC saves the most time. Done badly, it is where real attacks are closed by mistake.

The questions a good triage answers

Each type of alert has its own checks, but the same handful of questions sit behind most of them. The one most often skipped is what is at risk. A suspicious sign-in to a test server and the same sign-in to the system that runs payroll are not the same alert, even if the same rule raised them.

Context is what makes these questions answerable. A team that knows what each machine does, who owns it and which business processes it serves can triage far faster than one that has to find out every time.

A triage that answers the questions below has done its job, even if the answer to some is “we don’t know yet”:

• What happened, in plain words, and when?
• Which machine and which account are involved, and what are they for?
• Is this normal for that machine, that account or that team?
• What else happened on the same machine or account around the same time?
• Is anything involved, such as an address, file or domain, known to be bad?
• If this is real, what is at risk?

Ordering the queue by what matters

Many queues are sorted by the severity the detection tool assigned, then by time. Severity is a useful start, but it is set by whoever wrote the rule, with no knowledge of your company. A medium-severity alert on one of your most important systems can matter more than a high-severity one on a laptop in a test lab.

A better order weighs three things together: how likely the alert is to be a real attack, how important the systems and accounts involved are, and how long it has been waiting. Alerts that belong to the same activity should be grouped, so the team sees one attack rather than a dozen separate tickets.

Write the ordering rules down and review them. If the team keeps finding that alerts near the bottom of the queue turned out to matter, the order is wrong, not the analysts.

Verdicts and confidence

A triage should end in a clear verdict from a short, fixed list. A list that works well is: real attack, suspicious, harmless, authorised activity and inconclusive. Authorised activity covers alerts that are accurate but expected, such as an administrator running a tool that looks like an attacker’s. Inconclusive is an honest answer when the data needed to decide is missing.

Alongside the verdict, say how sure you are. A confidence level, whether a percentage or a simple high, medium or low, tells the next person how much checking the verdict needs. A harmless verdict at low confidence is a prompt to look again, not a reason to close.

Over time, compare confidence with outcomes. If verdicts marked as highly confident are often overturned on review, the team or the tool is overconfident, and that is worth knowing.

Writing the reason down

The verdict is only half the work. The reason is what lets someone else check it, and what makes the alert useful later. A good reason is short, specific and points to evidence: which records were checked and what they showed.

Compare two notes on the same alert. “Looks fine, closed” tells the next person nothing. “Backup job run by the backup service account, same time every night for the past month, copying to our own storage server” can be checked in a minute and reused the next time the alert fires.

Written reasons also show where rules need tuning. If the same reason appears on the same alert every week, the rule is asking a question the team already knows the answer to.

When to escalate

Escalation should be a rule, not a feeling. Agree in advance which situations go straight to a senior analyst or an on-call lead, so a tired person at the end of a shift does not have to make that call alone.

When you escalate, pass on what you found, not just the alert. The questions you asked, the answers and the records behind them save the next person from starting again.

Situations that usually justify escalating straight away:

• The verdict is real attack or suspicious, at any confidence.
• An important system, a privileged account or sensitive data is involved.
• The same activity appears on more than one machine or account.
• The data needed to decide is missing, or its source has gone quiet.
• Something does not fit and you cannot say why.

How armor360 handles triage

armor360 collects new alerts automatically and orders the queue by its verdict and the alert’s severity, real attacks first. A person can change an alert’s priority. Each alert gets a verdict of real attack, suspicious, harmless or not sure, with a confidence and the reason, backed by the records it collected. Verdicts show what is at risk in business terms, using what you told it about each machine during set-up.

In our lab, an alert for a burst of reconnaissance commands on a Linux machine was worked through 13 questions and 22 records from 4 data sources, and called a real attack at 90% confidence in under six minutes. A person then reviewed the verdict. armor360 recommends; people decide and act.