Why nights and weekends are hard to cover
To cover security alerts at night and at weekends, decide in advance which alerts must reach a person straight away, who that person is, what they are expected to do, and how the rest of the night’s alerts are handed to the day team in the morning. Most teams cannot staff a full shift through the night, so the work is in choosing well what can wait.
Attackers know when offices are empty. Activity that would be noticed at eleven in the morning can run for hours overnight or across a long weekend. At the same time, most alerts raised at night are no more urgent than those raised by day, and waking people for them wears out the people you most need.
Good out-of-hours cover balances those two facts. It keeps the urgent path short and clear, keeps the rest of the queue moving, and protects the on-call person’s sleep for the nights when it really matters.
Which on-call model fits your team
There are a few common ways to cover the hours outside the working day. Each has costs, and the right one depends on the size of the team, where its people are, and how much risk the business is willing to carry overnight.
Whichever you choose, write it down: who is on call this week, how they are reached, who backs them up if they do not answer, and how on-call time is recognised in pay or time off. On-call that is informal tends to fall on the same one or two people.
- On-call rota: one person carries a phone out of hours, on a weekly rotation, and is woken only for urgent alerts
- Follow-the-sun: teams in different time zones each cover their own daytime and hand the queue on at the end of it
- Outsourced night cover: an MSSP or MDR provider watches the alerts out of hours and calls your on-call person when needed
- Shift team: people working nights in turn, which needs enough staff to cover leave and sickness
- Mixed: an outside provider or an AI first look overnight, with an internal on-call person for escalation
What really needs to wake someone
The most important decision in out-of-hours cover is the threshold for a call. Set it too low and the on-call person is woken for noise, learns to distrust the pager and stops answering quickly. Set it too high and a real attack runs until morning.
Base the threshold on evidence and on business impact, not on rule severity alone. A high-severity label set by a rule author is a weak reason to wake someone. A finding that an attack is likely, on a system that matters, is a strong one. Write the threshold down and review it after each call-out: was it worth waking someone, and would anything have been lost by waiting?
Alerts below the threshold still need a home. Keep them in the queue with enough detail that the morning team can pick them up quickly, and make sure nothing is closed overnight without a reason recorded.
- Strong evidence of a real attack, such as a known-bad file or remote access tool on a server
- Suspicious activity on a system the business has marked as critical
- Signs of an attack moving between machines, or several stages on one machine
- Signs of data being gathered or sent out
- An important system going silent, where tampering cannot be ruled out
What an on-call runbook should include
The person woken at three in the morning is not at their best, and may not know the system involved. A short runbook for each kind of urgent alert turns that call into a set of steps rather than a puzzle. Keep it to one page, written in plain language, and kept where the on-call person can reach it from their phone.
A runbook should say what to check first, who owns the affected system, who to call next and what the on-call person is allowed to decide alone. Decisions that affect the business, such as taking a server offline, should name who can approve them out of hours. If that person cannot be reached, the runbook should say what happens instead.
Test the runbooks. Walk through a past incident with the people on the rota, or run a short exercise using an alert from a test machine. Gaps in contact details and access are much better found on a Tuesday afternoon.
- The alert types the runbook covers
- The first three checks, with where to find each
- System owners and business contacts, with numbers
- What the on-call person may decide alone, and what needs approval
- How to record what was done, and when
- How to hand over in the morning
How to run the morning handover
The morning handover is where night cover succeeds or fails. If the day team arrives to a long list of unread alerts, the night was not covered; it was postponed. A good handover tells them what happened, what was decided and what is waiting.
Keep the handover short and written. List any call-outs and what came of them, the alerts still open and how far each has got, and anything that needs an owner’s decision during the day. Then sort the overnight queue so that the most likely real attacks are read first, not the oldest alerts.
Use the handover to learn. Each week, look at what woke someone, what could have waited, and what should have woken someone but did not. Adjust the threshold, the rules and the runbooks from what you find.
How an AI first look changes the night
An AI investigator does not sleep, and it changes what the night looks like. Instead of alerts piling up unread until morning, each one can be investigated as it arrives: the agent searches the events around it, looks up the host and account, checks threat intelligence and writes a verdict with its evidence.
That changes two things. In the morning, the day team finds investigated work waiting rather than a raw queue: verdicts to review, with the evidence attached, sorted with the likely real attacks first. And at night, urgent findings can be sent to the on-call person with the investigation already done, so they are woken with a reason and a starting point rather than an alert title.
It does not change who decides. A person still reviews the verdicts and makes the calls that affect the business. The AI takes the first look; people review it and act. When choosing a tool for this, check what it does on its own overnight. For most organisations, the safe answer is that it investigates and tells people, and takes no action on systems.
How armor360 approaches it
armor360 investigates new alerts as they arrive, gives each a verdict with its confidence and evidence, and queues likely real attacks first. During set-up, the company tells it which systems are important and who to call for urgent findings, and findings can be sent to Slack, Microsoft Teams, PagerDuty or email.
In the morning, the Today page opens on what is waiting for a person. armor360 takes no response actions: it recommends, and people decide and act.