Start with what raised it
Every investigation starts with the same question: why did this alert fire? An alert is a rule’s opinion that something matched a pattern. Before you can judge whether the pattern means harm, you need to know what the rule was looking for, which event matched, and how often that rule fires on an ordinary day.
Open the rule itself, not only the alert title. A title such as “Suspicious PowerShell” can cover a dozen different behaviours. The rule’s query tells you exactly what was seen: a command line, a file write, a network connection, a sign-in from a new place. Note the time of the triggering event as precisely as you can, because every later question is asked around that moment.
It also helps to know the rule’s history. A rule that fires many times a week on the same administrator’s scripts deserves a different first glance from one that has never fired before. History does not decide the case, but it tells you where to start looking.
Identify the machine and the person
Next, work out what the alert happened to. Which machine raised it, what operating system it runs, where it sits on the network and who looks after it. Then work out who was involved: the account that ran the process or signed in, whether it belongs to a person or a service, and what that account normally does.
This is where many alerts settle quickly. A backup job running as a service account at two in the morning looks alarming on its own and ordinary once you know the account’s purpose. A developer’s account opening a shell on a payroll server is the reverse: ordinary on its own, worth a closer look in context.
- Host name, address and operating system
- The owner or team responsible for the machine
- The account involved, and whether it is a person or a service
- What that account usually does, and when
- Recent changes: new software, new access, a new owner
Look at what happened nearby
An alert is one event. The story is in the events around it. Search the minutes before and after the triggering event, on the same machine and for the same account. Which processes started? Which files were written? Which other machines did it talk to? Did the same account log on anywhere else?
Process ancestry is often the most telling part. Every process has a parent, and the chain of parents shows how the activity began. A command shell started by a web server, a script launched by an office document, or a tool run from a temporary folder by an unexpected parent are all signs that something reached the machine from outside. A command started by a known management agent, under its usual account, usually is not.
Look for other alerts on the same machine or account over the past few days too. A single odd event can be noise. Several odd events on one machine, each from a different rule, are much harder to explain away.
Check what is already known to be bad
Most alerts carry indicators: IP addresses, domain names, file hashes, web addresses. Look each one up in the threat intelligence you have, whether that is a shared feed, a sandbox report or an internal list of addresses you have seen before. A match on a known malicious address is strong evidence. A clean result is weaker evidence than it seems, because new attacker infrastructure has no history yet.
Treat reputation as one input among several. A known-bad file hash on a machine settles a lot. An unknown domain contacted on a regular schedule, with no business reason, can matter just as much even though no feed has flagged it.
Ask what the machine is for
The same activity means different things on different machines. A network scan from the security team’s test server is expected. The same scan from a till in a shop is not. Before you settle on a verdict, find out what the machine does for the business, which processes depend on it and how much harm a compromise would cause.
This context changes both the verdict and its urgency. A suspicious finding on a machine that runs end-of-day settlement needs a person to look at it now. The same finding on a lab machine due to be rebuilt can wait until morning. Writing this context down once, for every important system, saves each analyst from rediscovering it in the middle of the night.
Keep the evidence and write a verdict others can check
An investigation is only as good as its record. Keep each search you ran and what it returned, each lookup you made and the answer you got. When you write the verdict, point to the specific records that support it. Someone reading it later should be able to follow your reasoning without repeating your work.
A good verdict makes a clear call: real attack, suspicious, harmless, or not enough evidence to say. “Inconclusive” is an honest answer when the data is missing, as long as you say which data would settle it.
Write for the person who has to act on it, not only for other analysts. A manager deciding whether to take a server offline needs the risk in business terms, not a list of process identifiers.
- The call, in plain words
- How sure you are, and why
- The records that support it, cited
- Recommended next steps, and who should be told
A worked example from our lab
In our lab, MITRE Caldera, an adversary simulation tool, runs against a few Linux machines watched by Elastic Security. One of its agents ran a burst of reconnaissance commands on a machine called linux04, and Elastic raised an alert titled “Burst of reconnaissance commands”.
armor360 worked the alert through the questions above. It asked thirteen questions, read four data sources and kept twenty-two records as evidence. Looking at what ran nearby, it found a process named caldera-agent calling out over HTTP every five minutes, and tied the reconnaissance commands to it. The verdict was real attack, at 90% confidence, reached in 5 minutes 35 seconds. A person then reviewed it and decided what to do.
The point is not the speed. It is that every step was written down, so the person reviewing the verdict could check each claim against the record it came from.