What was the AI allowed to do?
Auditors rarely ask how an AI model works inside. They ask about control: what the system could do, who decided that, and how you know it stayed within those limits. If you use AI in security operations, expect five questions, and prepare an answer and a piece of evidence for each. This one comes first because the others depend on it.
A good answer is a short, written scope. Which data sources the AI can read. Whether it can change anything: close tickets, switch rules on or off, act on machines or accounts. Which credentials it uses and what permissions those credentials have. An auditor will compare that document with the real configuration, so write down what is true, not what was intended.
Check the scope against reality yourself before an auditor does. Log on with the credentials the AI uses, or ask your administrator to list their effective permissions, and confirm they match the document. Permissions tend to grow over time as someone adds “just one more” connection, and an auditor who finds an undocumented write permission will rightly ask what else is undocumented.
The easiest scope to defend is read-only. An AI that can search events, read asset records and look up indicators, but cannot change anything, carries a much smaller risk than one that can. If yours can take actions, list each one, the conditions under which it may, and the approval it needs.
- A list of connections and the permissions of each credential.
- A plain statement of what the AI can change, if anything.
- Where that scope was approved, and by whom.
Who approved what, and when?
The second question is about accountability. When the AI said an alert was harmless, did a person agree? When a detection rule went live, who approved it? When a setting changed, who changed it and was a second person involved? Auditors want to see that people remain responsible for decisions, and that the record shows which person.
Prepare an export of the audit trail for a sample period and check it yourself first. Look for gaps: shared accounts that hide who acted, changes without a reason, approvals given by the same person who requested them. Fix those before the audit rather than explaining them during it.
Sensitive changes deserve stronger controls, and an auditor will look for them. Switching off a detection rule is a good example. A rule turned off quietly can blind the team for months, so requiring two administrators for that change is the kind of control that reads well in an audit report.
Can you show the evidence?
Expect the auditor to pick an alert from months ago and ask why it was closed. A verdict on its own is not an answer. The answer is the records that led to it: the events searched, the asset details, the related alerts, the threat-intelligence lookups, and the reasoning that connects them.
Test this before the audit. Choose three closed alerts at random, one marked harmless, one suspicious and one a real attack, and see whether you can show the full working for each within a few minutes. If the records were not kept, or only a summary was, the answer to “can you show the evidence?” is no.
Check retention too. If your policy says investigation records are kept for a year, confirm that records from eleven months ago are still there and readable.
How are mistakes caught?
Any analyst, human or AI, will sometimes be wrong. The auditor wants to know how you find out. The weakest answer is “we trust it”. A strong answer describes several layers.
Bring numbers from your own records: how many verdicts were changed by a reviewer last quarter, in which direction, and what was done about the pattern. A rising rate of changes is not a failure if you noticed it and acted. Not knowing the rate is.
Simulated attacks deserve a special mention. Running a known, harmless attack against your own systems, and checking that it was noticed and called a real attack, is direct proof that the chain from log to verdict works. Keep the results, including the times it did not work and what you changed.
- A person reviews verdicts, and can agree, change them or add context.
- Changes are recorded, and the disagreement rate is tracked over time.
- Verdicts with low confidence go to a person rather than being taken as settled.
- Simulated attacks are run against your own systems to check they are noticed and called correctly.
- Corrections are fed back, so the same mistake is less likely next time.
Where does the data go?
Security alerts carry names of people and machines, internal addresses and details of what systems were doing. Auditors will ask where that data travels when an AI reads it. Draw the flow on one page: where alerts come from, where the AI runs, which outside services receive anything, and what they receive.
Be precise about the AI model. Does it run on your own hardware, in your private cloud or at a provider? If it runs locally, is there a fallback to an outside model when the local one is busy, and is that switched off? Then list the other outbound connections. Threat-intelligence lookups send indicators such as addresses and file hashes to outside services, and notifications send summaries to chat and ticketing tools. None of that is wrong, but all of it should be on the page.
Add retention and access to the same page. How long are alerts, investigation records and AI working notes kept? Who can read them? If the AI provider can see your data for support or improvement, say so and say under what agreement. Auditors are rarely troubled by a well-described flow; they are troubled by one nobody can describe.
How to prepare the answers
Put the answers in a small evidence pack and keep it current, rather than assembling it under pressure each year. Each item should be something you can show, not a statement of intent. Review it when you add a connection or change a setting, and the audit becomes a walk-through rather than a search.
Name one owner for the pack. In a small team that may be you. The owner does not have to write everything, but they check it each quarter and before each audit, and they know where each piece came from. When the auditor asks a follow-up question, the owner can answer it or knows who can.
- The written scope of the AI, matched to its real configuration.
- An audit-trail export for a recent period, checked for gaps.
- Three sample investigations with their full records.
- Your review process, and last quarter’s rate of changed verdicts.
- The data-flow page, including the model’s location and outbound connections.
Where armor360 fits
armor360 investigates and recommends; it takes no response actions, which keeps its scope simple to describe. Every record it collects is kept and cited where it is used, people review verdicts, switching a rule off needs two administrators, and the audit trail keeps who approved what and when.
It can run fully on-premises, our own model included, with external AI fallback switched off, so alert data need not leave your environment.
