Why test a rule first
A new detection rule is a guess about what attacker behaviour looks like in your data. Put live without testing, it can fail in two ways. It can fire constantly on ordinary activity, burying real alerts and wearing down the people who read them. Or it can never fire at all, because the data it depends on is not arriving or the fields are named differently from what the rule expects.
Both failures are cheap to find before a rule goes live and expensive to find afterwards. Testing a rule against data you already hold tells you how it will behave on a normal day, and gives the person approving it something concrete to look at.
Testing also sets expectations. If a rule is known to fire a few times a week, mostly on one team’s tooling, the people reading its alerts can be told so in advance, and a sudden change in its volume becomes a signal in itself.
Check the data is ready
Every rule depends on specific data: process events from endpoints, authentication logs, DNS queries, cloud audit logs. Before anything else, confirm that the data is arriving, from the machines the rule is meant to cover, with the fields the rule uses.
It helps to sort a rule library by readiness. Rules whose data is already arriving can be tested and switched on. Rules that need more data point to collection you should add. Rules that depend on sources you do not collect at all can wait. Sorting this way also turns detection planning into a list of concrete collection tasks.
Check field names as well as presence. The same information can arrive under different names from different collectors, and a rule written for one will match nothing from the other.
- Ready now: the data is arriving and the fields match
- Needs more data: some sources or machines are missing
- Locked: the source the rule depends on is not collected
Run it against the past
The most useful test is to ask how often the rule would have fired on data you already have. Run its query over a recent period of stored events or past alerts, count the matches, then read a sample of them.
If it would have fired hundreds of times a day, almost all on routine activity, it is not ready. If it would never have fired, check whether that is because the behaviour is absent or because the data is. If it would have fired a handful of times, read each one: were any of them real, and would the alert have told an analyst enough to act on?
Where you can, test against known attack activity too. Running a simulated technique in a test environment shows whether the rule catches what it was written for.
Write down what you found and keep it with the rule. The next person to tune it will want to know what it was tested against, and what the results looked like.
Would have fired 473 times
Would have fired 3 times
Dealing with noisy rules
Noise is the most common reason detection rules lose trust. A rule that fires often on the same harmless activity teaches people to close its alerts without reading them, and the one real alert gets closed with the rest.
Tune noise with care. Narrow exceptions, such as a specific service account running a specific script on specific servers, are safer than broad ones. Excluding a whole folder or a whole family of accounts can open a gap an attacker can walk through. Record each exception and why it was added, so it can be reviewed later.
Some rules work better as scheduled hunts. If a behaviour is worth looking for but too common to alert on in real time, a weekly hunt whose results a person reads can cover it without the noise.
Approval and change control
Detection rules are part of your security controls, and changes to them deserve the same care as changes to a firewall. A rule going live should be approved by a person who has seen the test results. Switching a rule off, or adding a broad exception, deserves more scrutiny still, because it removes protection quietly.
A simple pattern works well: one approver to switch a tested rule on, two administrators to switch a rule off. Keep an audit trail of who approved what and when, with the reason. When an auditor or an incident review asks why a rule was not firing on a given date, the answer should already be on record.
Measuring ATT&CK coverage
Mapping each live rule to the MITRE ATT&CK techniques it detects shows where coverage is thick and where it is thin. A coverage report is most useful when it counts only rules that are live and whose data is arriving. A rule for a technique that cannot fire, because its data is missing, is not coverage.
Use the report to plan, not to reassure. Thin areas that matter to your environment, such as credential access on the systems that hold your most important data, are where the next rules or hunts should go.
Coverage reports also help in conversations with auditors and budget holders, as long as they are honest. Showing where you cannot see, and what it would take to see there, is more useful than a single percentage.
How armor360 approaches it
armor360 sorts its rule library by whether the data each rule needs is already arriving: ready now, needs more data, or locked. Before a rule goes live, it shows how often the rule would have fired on stored past alerts.
When a person asks it to draft a rule, the agent writes one from what it has learned in investigations. The person can narrow or edit it, and a new rule needs approval before it goes live. Switching a rule off needs two administrators, and an ATT&CK coverage report shows what the live rules cover.