How to run an AI SOC pilot on your own alerts

What an AI SOC pilot should prove

An AI SOC pilot should answer one question: on your own alerts, in your own environment, does the platform reach verdicts your analysts agree with, quickly enough and clearly enough to change how your team spends its time? Everything in the pilot plan exists to answer that question with evidence rather than impressions.

That rules out some common shortcuts. A pilot that runs on the vendor’s sample data proves the product works on the vendor’s data. A pilot with no agreed measures ends in a vote on whether people liked it. A pilot where nobody reviews the verdicts produces a pile of output and no conclusion. The work is not difficult, but it has to be planned before the first alert arrives.

Be realistic about what a pilot cannot prove. A few weeks of alerts will not include every kind of attack, and it may not include a real one at all. That is why known-answer cases matter: they let you test the cases you care about most without waiting for them to happen.

How to set scope and success criteria up front

Agree the scope in writing before you connect anything. Which alert sources are in: one SIEM, an EDR, both? Which parts of the estate: the whole organisation, one business unit, a set of servers? Which kinds of alert: all of them, or the high-volume rules that take most of your team’s time? A narrow, well-chosen scope gives clearer results than a broad, shallow one.

Then agree what success looks like, with numbers your team picks before seeing any results. For example: the platform agrees with reviewers on a stated share of harmless alerts; it misses none of the known attacks; a reviewer can check a verdict in a stated number of minutes; it says it is unsure rather than guessing when data is missing. Writing these down first stops anyone, vendor or buyer, from moving the goalposts later.

Agree the exit too. Decide who makes the final call, what happens to the data at the end, and how the platform will be disconnected if you do not go ahead.

• Alert sources, parts of the estate and alert types in scope
• Success criteria with thresholds, agreed before results arrive
• A named decision-maker and a decision date
• What happens to pilot data afterwards, and how access is removed

Which alerts to include in the pilot

Use two streams of alerts. The first is live: new alerts as they arrive during the pilot, so you see how the platform copes with your real daily mix. The second is a set of known-answer cases taken from your history, where your team already knows the right verdict.

Build the known-answer set with care, because it carries a lot of the evaluation. Include confirmed attacks or red-team exercises if you have them, harmless alerts with a recorded cause, authorised activity such as a penetration test or a planned change, and a few alerts your team itself found hard. If you can, run a small, safe simulation during the pilot, such as an adversary-emulation tool on a test machine, so there is at least one fresh attack with a known answer.

Do not hand-pick only easy cases, and do not tell the platform the answers. The aim is to see how it behaves on the alerts that actually take your team’s time, including the messy ones with missing data.

Who reviews, and how to connect read-only

Name the reviewers before the pilot starts and give them protected time. Two or three analysts who know your environment are enough. Ask them to review the platform’s verdicts the way they would review a colleague’s: read the reasoning, check the cited evidence, and record whether they agree, disagree or cannot tell, with a sentence on why. Their records are the pilot’s main result, so make recording easy.

Connect with the least access that lets the platform do its work. For an investigation-only pilot that means read access: collecting alerts, searching logs and reading asset information. It should not need rights to change rules, alter data or act on machines. Use a dedicated service account so that everything the platform does is attributable and easy to remove afterwards.

Tell the people who need to know: the owners of the data sources, whoever looks after the service accounts, and your data protection lead if alert data will be processed somewhere new.

What to measure during an AI SOC pilot

Measure a small number of things well. The most important is agreement with your reviewers, split by verdict type, because agreeing on harmless alerts and agreeing on attacks are different achievements. Alongside it, measure how long the platform takes to reach a verdict, how long a person spends reviewing each one, and how often the platform says it is unsure.

Review time is the measure most often left out and the one that matters most for the business case. A fast verdict that takes a reviewer as long to check as the original investigation would have taken saves nothing. A verdict a reviewer can confirm in a couple of minutes because the evidence is laid out and cited is where the value is.

• Agreement rate with reviewers, separately for attacks, suspicious, harmless and authorised
• Known attacks called harmless: the number that should be zero
• Time from alert to verdict
• Time a person spends reviewing each verdict
• Unsure rate, and whether “unsure” came with a clear reason
• Verdicts where the cited evidence did not support the claim

A two-to-four-week pilot plan, step by step

A focused pilot fits in two to four weeks. The shorter end suits a single alert source and a small scope; the longer end suits several sources or a provider testing more than one customer. Whatever the length, keep weekly check-ins short and based on the reviewers’ records.

Front-load the known-answer cases. Running them in the first week gives reviewers a shared sense of how to record agreement, shows early whether the connection and context are right, and leaves the remaining weeks for live alerts. If something basic is wrong, such as a log source the platform cannot search, you find out while there is still time to fix it.

• Before day one: agree scope, success criteria, reviewers and the decision date
• Days 1 to 3: create read-only service accounts, connect alert sources and log search, and record what each machine does for the business
• Days 3 to 5: run the known-answer set and review it together, to calibrate how reviewers record agreement
• Week 2: let live alerts flow; reviewers work through the queue daily and record each verdict
• Weeks 2 to 3: correct verdicts where reviewers disagree, and check whether the corrections carry into later investigations
• Optional: run a safe attack simulation on a test machine and follow the investigation
• Final week: compile the measures, read the disagreements, and hold the decision meeting

How to read the results honestly

Start with the misses, not the averages. Read each case where the platform called a known attack harmless, or where the evidence it cited did not support its claim. One serious miss with high confidence matters more than a good overall agreement rate, and the reason behind it tells you whether it is fixable: missing data, a misread record, or a reasoning error.

Then look at whether reviewers were right. Pilots often find that a disagreement was the reviewer’s mistake, or that both readings were defensible. Count those separately. Finally, compare the review time per verdict with how long the same alerts took your team before, and turn that into hours per week at your real volume. Present ranges, not single figures, and state the assumptions.

Most disappointing pilots fail for reasons that have little to do with the product. These are the pitfalls to avoid.

• Judging the pilot on the vendor’s data rather than yours
• Setting success criteria after seeing results
• Reviewers with no protected time, who skim or skip
• A known-answer set made only of easy cases
• Counting agreement without looking at the misses
• Granting broad write access “to save time”

How armor360 approaches it

armor360 offers a free pilot on your own alerts. It connects to Elastic Security, Splunk, Microsoft Sentinel, CrowdStrike Falcon or Microsoft Defender, collects new alerts and gives each a verdict with a confidence, the reason and the cited records behind it. Reviewers agree with a verdict or change it and add context, and corrections are remembered for later investigations.

It takes no response actions, so a pilot is about investigation and verdicts: it recommends, and your people decide and act. In our lab, a reconnaissance alert from a MITRE Caldera agent on a machine called linux04 was worked through 13 questions and 22 records from 4 data sources, and called a real attack at 90% confidence in 5 minutes 35 seconds.