What suspicious process activity on Linux looks like
To investigate suspicious process activity on Linux, find out what ran, what started it, which account ran it, what it talked to on the network, and how it survives a restart. Most attacks on Linux servers leave traces in those five places, and the commands attackers run are usually ordinary system tools used in an unusual order, by an unusual account, or from an unusual parent.
That last point is what makes Linux investigation hard. There is rarely a single malicious file to find. An attacker with a foothold runs the same commands an administrator would: ps, cat, grep, whoami, uname, id, ls. The difference is in context. Who ran them, how quickly, from which parent process, and what happened next.
This guide walks through the questions to ask, the logs you need to answer them, and a worked example from our lab.
How to recognise reconnaissance commands
After gaining access, an attacker needs to learn where they are. On Linux this usually means a burst of short discovery commands, run within seconds or minutes of each other. Detection rules often look for exactly this: several discovery commands from one process tree in a short window.
Each command on its own is harmless and common. Administrators run whoami and uname all day. What matters is the pattern: many discovery commands in quick succession, from a process that is not a person’s interactive shell, or from an account that normally runs one service and nothing else.
- whoami and id: which account am I, and which groups does it belong to?
- uname -a and cat /etc/os-release: which kernel and distribution is this?
- ps aux: which processes are running, and is there security software?
- cat /etc/passwd and cat /etc/group: which accounts exist?
- grep across configuration files: are there passwords, keys or connection strings?
- ip addr, ss or netstat: which networks and services can this machine reach?
- ls of home folders and .ssh directories: are there keys to reuse elsewhere?
How to read parent and child process chains
Every Linux process has a parent, and the chain of parents shows how the activity began. Build the chain upwards from the suspicious command until you reach something you recognise: systemd, sshd, cron, a web server or a known management agent.
Some chains are ordinary. sshd starting bash for a named administrator, who then runs ps, fits a person logging in to check something. Others are warning signs. A web server process such as nginx, apache2 or a Java application server starting /bin/sh, which then runs whoami, suggests that someone is running commands through a web vulnerability. A shell started from a process in /tmp or /dev/shm, or from a binary with a random name, deserves close attention.
Look at the command line of each process in the chain, not only the name. Attackers often rename binaries to look like system processes, and the full path and arguments give the game away: a process called “kworker” running from a home folder is not a kernel thread.
How to check cron jobs and scheduled scripts
Attackers who want to keep access add a way to come back after a restart or after their session ends. On Linux the usual places are cron and systemd. Check the system crontab, the files in /etc/cron.d and the cron.hourly, cron.daily and similar folders, and each user’s crontab. Then check systemd units in /etc/systemd/system and in users’ home folders, along with shell start-up files such as .bashrc and .profile.
Look for entries created recently, entries that run scripts from temporary or hidden folders, entries that download something with curl or wget and pipe it to a shell, and entries owned by accounts that should not be scheduling anything. Compare against a known-good copy of the machine if you have one, or against a sibling server built the same way.
File events help here. If your endpoint data records file writes, search for changes to cron and systemd paths around the time of the alert. A new cron entry written a few minutes after a burst of reconnaissance is strong evidence that the two belong together.
How to spot beaconing and misused interpreters
Beaconing is a program calling out to the same destination on a regular schedule, waiting for instructions. It is how many remote access tools keep in touch with whoever controls them. In network or process connection data it shows up as repeated connections from one process to one address or domain, at near-fixed intervals, often over HTTP or HTTPS so that it blends in with ordinary web traffic.
To find it, group outbound connections by process and destination and look at the times between them. Regular gaps, a destination with no business purpose, and a process that is not a browser or a known updater are a strong combination. Attackers often add small random delays, so look for near-regular rather than exactly regular timing.
Interpreters deserve particular attention. python3, perl, bash and similar tools can run anything, and attackers use them because they are already installed. python3 running under a service account such as www-data, a database account or an application user, with a command line that opens a socket or runs code passed in on the command line, is worth investigating even if no rule has a specific name for it. Ask whether that account has any reason to run an interpreter at all.
What logs you need to investigate Linux activity
You can only answer these questions if the data is there. Many Linux servers send only their system logs, which record sign-ins and service messages but not the processes that ran. That is not enough to build a process chain or tie a network connection to the program that made it.
Check what you collect before you need it. An investigation that stops at “the logs do not show which process made this connection” is a sign of a collection gap to fix, not a dead end to accept.
- Process events with parent, command line, user and working directory, from Elastic Defend, Elastic Agent with auditd data, or another endpoint sensor
- Network events that tie each connection to a process, so beaconing can be traced to its source
- Authentication logs: SSH sign-ins, sudo use and failed attempts
- File events for cron, systemd, start-up files and temporary folders
- DNS queries, from the host or your resolvers, to see which names it looked up
- An inventory note on what the server is for and which accounts should run on it
A worked example from our lab
In our lab, MITRE Caldera, an adversary simulation tool, runs against Linux machines watched by Elastic Security. On a machine called linux04, a Caldera agent ran a burst of reconnaissance commands, and Elastic raised an alert titled “Burst of reconnaissance commands”.
armor360 investigated it with the questions in this guide. It asked thirteen questions, read four data sources and kept twenty-two records as evidence. Following the process activity nearby, it found a process named caldera-agent beaconing over HTTP every five minutes and tied the reconnaissance to it. The verdict was real attack, at 90% confidence, reached in 5 minutes 35 seconds. A person then reviewed the verdict and decided what to do.
The useful lesson is the order of the work. The reconnaissance alert alone could have been an administrator checking a server. The regular outbound calls from an unfamiliar process are what turned it into a clear finding, and that needed process and network data from the same host.
How armor360 approaches it
armor360 investigates alerts the way this guide describes: it asks itself questions and answers each by querying your data, including events in the SIEM, process and parent chains, other alerts on the host, user activity and threat intelligence lookups. Its investigation map shows each question, the data sources it read and the path of evidence behind the verdict.
It also finds machines that have stopped sending logs and suggests the Elastic integrations each machine needs, adding them through Elastic Fleet once a person confirms. armor360 recommends next steps; people decide and act.