How long should an AI SOC evaluation take?

Long enough to see the platform judge a realistic mix of your own alerts, including some with known answers, and to have your analysts review its verdicts. For most teams that means a structured pilot of a few weeks rather than a single demo. The time goes on agreeing criteria, connecting data and reviewing verdicts, not on watching the product. Plan the reviewer time in advance, because without it the evaluation has nothing to measure against.

What accuracy should I expect from an AI SOC?

There is no figure that holds across organisations, because accuracy depends on your alerts, your logging and how much context the platform has. Treat any single headline number with caution unless you know the data it came from. Measure it yourself instead: run known-answer cases, compare verdicts with your team’s, and look closely at the disagreements, especially real attacks called harmless.

Should an AI SOC take response actions automatically?

That is a choice for your organisation, and it should follow evidence of accuracy rather than come first. Automatic actions on a wrong verdict can disrupt the business, so many teams start with a platform that recommends and a person who decides. If you do want automated actions, check who approves them, whether approval can be required for each one, and how each action is recorded.

Is per-alert pricing a bad idea?

Not always, but it ties your bill to your noisiest rules and to the busiest days, such as during an attack. It can also discourage teams from switching on useful detections. If you are offered per-alert pricing, model the cost at double your current alert volume and ask what happens when a rule misfires for a day.

How to evaluate an AI SOC platform

What to test first: accuracy on your own alerts

Evaluate an AI SOC platform on your own alerts, in your own environment, against answers you already know. A demo on the vendor’s data shows you what the product looks like. It does not show you how the product judges the alerts your rules raise, on the machines you run, with the gaps your logging has. Accuracy is the first thing to test because every other feature depends on it.

Start by building a small set of known-answer cases. Take alerts your team has already closed with confidence: a few confirmed attacks or red-team exercises, a larger number of harmless alerts with a clear explanation (a backup job, an administrator’s script, a vulnerability scanner), and some that were authorised activity such as a planned penetration test. Feed them to the platform without telling it the answer, and compare its verdicts with yours.

Look at the disagreements one by one rather than only counting them. A platform that calls a harmless scanner “suspicious” and explains why is less worrying than one that calls a real attack harmless with high confidence. Missing a real attack and crying wolf do not cost the same, so weigh them differently. Note too how often the platform says it is unsure. A tool that never admits doubt is guessing some of the time.

Confirmed attacks or red-team activity your team has already worked
Harmless alerts with a known, recorded cause
Authorised activity: planned tests, change windows, approved tools
Alerts where your own team was unsure, to see how the platform handles doubt
Alerts from each source you run, not only the best-logged one

How to check that it shows its working

A verdict without its working is an opinion. For each investigation, you should be able to see which questions the platform asked, which searches it ran against your data, what each search returned, and which of those records it relied on for the verdict. If you cannot follow the chain from the alert to the conclusion, you cannot check it, and neither can an auditor or a customer.

Test this directly. Pick a verdict and try to verify one claim in it. If the platform says the process was started by a web server, can you click through to the record that shows the parent process? If it says an IP address is known to be malicious, can you see which threat intelligence source said so, and when? A good platform cites the record where the claim is made, not in a general list at the bottom.

Also check what the platform does when its working is incomplete. If a search timed out or a data source was missing, the investigation should say so plainly. Silent gaps are worse than visible ones, because the reader assumes the evidence was looked at when it was not.

Two draft rules, tested on the last 30 daysIllustrated example

Any archive created in /tmpToo noisy

Would have fired 473 times

Archive created, then copied to a staging folderReady to approve

Would have fired 3 times

Before a rule goes live, armor360 shows how often it would have fired on alerts you already store. The noisy one goes back for narrowing; the quiet one goes to a person to approve.

What its confidence scores should mean

Most platforms attach a confidence to each verdict. The number is only useful if it is calibrated: verdicts given high confidence should be right more often than verdicts given low confidence. A score that is always high, or that does not move when the evidence is thin, tells you nothing.

You can check calibration roughly during an evaluation. Group the platform’s verdicts on your known-answer cases by confidence band, and see how often each band agreed with your team. If the high-confidence band is wrong as often as the low one, the score is decoration. Ask the vendor how the score is produced, and whether it drops when data is missing.

Calibration matters because it decides how you use the tool. If high-confidence harmless verdicts are reliable, your team can review them quickly and spend its time on the uncertain ones. If they are not, every verdict needs the same careful read, and much of the time saving disappears.

Where your data goes and who can see it

An AI SOC reads some of the most sensitive data you hold: sign-in records, command lines, file names, email subjects, internal addresses. Before you look at features, find out where that data is processed and stored, and by whom. Ask whether the AI model runs inside your environment, in the vendor’s cloud, or with a third-party model provider, and whether any of that can be switched off.

Ask the plain questions in writing. Is alert data used to train models shared with other customers? How long are investigation records kept, and can you delete them? Who at the vendor can see your data, and is that access logged? Can you choose the region where it is processed? The answers belong in your contract, not only in a sales call.

Where the AI model runs, and whether an external fallback can be switched off
Whether your data trains anything other customers use
Retention periods for alerts, evidence and reports, and how deletion works
Vendor staff access to your data, and how it is logged
How sign-in works: single sign-on, a second step, re-confirmation for sensitive changes

How it fits your tools and what it does on its own

Check the integrations against what you actually run, not against a logo wall. The platform needs to collect alerts from your SIEM or EDR, search the underlying logs to investigate, look up indicators in your threat intelligence, and send findings where your people work: chat, paging, email or a ticketing system. Ask which of these are built and in use today and which are on a roadmap. An integration that only collects alerts, without searching the logs behind them, gives the AI little to investigate with.

Then be clear about what the platform does without a person. Some products only investigate and recommend. Others can take response actions, such as isolating a machine or disabling an account. Decide which you want before the demo, and check that the controls match: who approves an action, whether approval can be required, and how actions are recorded. A tool that acts on its own needs much stronger evidence of accuracy than one that only advises.

Ask how the platform learns from your corrections. When an analyst changes a verdict and explains why, does that knowledge carry into later investigations for your organisation, and only yours? Can you see what it has remembered, and remove something that turns out to be wrong? A platform that makes the same mistake on the same alert every week costs you review time indefinitely.

If you are a managed security provider, test multi-tenancy properly. Each customer should have its own separate data, users, connections, settings and audit trail, and an analyst’s correction for one customer should not change verdicts for another.

How pricing models change what you pay

AI SOC platforms are usually priced in one of three ways: per alert or per investigation, per seat (per analyst or user), or by the size of what you protect, such as machines, users or data volume. Each one shapes behaviour, so think about how yours will grow before you compare quotes.

Per-alert pricing is easy to understand but makes cost follow your noisiest rules. A new detection that fires often, or an attack that triggers a flood of alerts, raises the bill at the worst moment, and it can quietly encourage teams to turn off useful rules. Per-seat pricing discourages giving access to the people who should see findings, such as system owners, managers and auditors. Pricing by the size of what you protect is steadier and keeps alert volume and access out of the sum, but make sure you understand how size is counted and what happens as you grow.

Whatever the model, ask for the full cost at your current volume and at double it, what is included (model running costs, storage, integrations, support), and whether there are limits that trigger extra charges. Compare that against the analyst time the platform realistically saves on your known-answer tests, not the vendor’s headline figure.

Red flags to watch for in a demo

Demos are rehearsed, which is fine, but some patterns should make you ask harder questions. The most common is a demo that only ever shows real attacks being caught. Most of a SOC’s work is deciding that alerts are harmless, so ask to see harmless and uncertain verdicts too, and how the platform explains them.

Be wary of answers that avoid the specific. If you ask how a verdict was reached and the reply is about the model’s general ability, rather than the records behind this verdict, the working may not be visible. The same applies to accuracy claims with no method behind them. Ask what data the figure came from and whether you can repeat the test on your own alerts.

Only attacks are shown, never harmless or unsure verdicts
Accuracy figures with no data set or method you can check
Confidence scores that are high on almost everything
Evidence summarised in prose, with no link to the underlying records
Reluctance to run a pilot on your alerts, in your environment
Integrations described as available that turn out to be planned
No clear answer on where the AI model runs or what it is trained on

How armor360 approaches it

armor360 is an AI SOC analyst. It collects alerts from Elastic Security, Splunk, Microsoft Sentinel, CrowdStrike Falcon and Microsoft Defender, investigates each one by querying your data, and gives a verdict of real attack, suspicious, harmless, authorised or unsure, with a confidence and the reason. Every record it collects is kept and cited where it is used, and the investigation map shows the path that backs the verdict. It takes no response actions: it recommends, and people decide and act.

It runs on its own AI model, fine-tuned by our team, and can run fully on your own hardware, the AI included; external AI fallback can be switched off. Corrections are remembered for later investigations about the same company, each customer of a provider gets its own tenant, and pricing is by quote, sized to what you protect, with no charge per person.