How do you audit your Linux environment? How do you track after changes to your files? What kind of processes are running on your system at any given moment? What uses the most resources? Valid questions, all. Special contributor Dedoimedo gives us the straight scoop on “audit.”. Editor’s note: Call for submissions: are you an OS expert? Can you provide some special insight, some tips and tricks, or just plain illuminate an obscure feature in your OS of choice? We’d like to publish it.
We’ve monitored system activity with sar. We used iostat, vmstat and dstat to collect statistics
on resource usage. We learned about pacct for process accounting. We also worked with top in
batch mode and atop. Finally, we worked with some high-end debugging tools like OProfile and
strace. Links below. But we did not use any utility for file auditing.
Today, we do that. The name of the game: Linux audit.
What Linux audit be?
Linux audit is a built-in mechanism in the kernel, which allows you to intercept changes to
monitored files and write them to a log on the disk. The mechanism consists of three major
components: auditd, the audit daemon (background service),
auditctl, the program used to control the auditing system, and
ausearch, a tool to query audit daemon logs.
To get audit configured, you will need to install the audit (or auditd) package, if it’s not
already installed. Once installed, you will probably want to setup audit to run on startup,
using a system command-line tool like chkconfig, for instance.
You can also manually control the service via /etc/init.d.
Why not use version control?
That’s a good question. The big difference between audit and version control is that the
first is mandatory. Version control works only if people want to cooperate. Let’s see an
Take subversion (svn), for instance. You can have all kinds of files and directories under
version control. Indeed, every time you update your files from the central repository, you
will override any local changes done to your files. While svn blame will tell you about file
edits against the subversion server, it won’t tell you anything about local changes.
Furthermore, this requires another layer of setup.
An alternative is configuration management, like cfengine. This could work, too. You will
have a static baseline to revert to, deleting any unwanted changes to your files. However,
you will not know, in between period runs, who made changes to your files – or why.
Both version control and configuration management are excellent resources, but they rely on
good will. They work extremely well in an environment where you have cooperative users
working toward a common goal. If you merely want to police your files against all and any
desired and undesired changes, then you want auditing.
So auditing it shall be.
Audit in action (auditctl)
Let’s see how this thing works. First, let’s make sure the service is running:
Now, we can start working on collecting audits. You can configure auditing rules on the fly
via auditctl, plus you can set some defaults using the configuration files.
/etc/audit/auditd.conf is used to configure log file location,
size and flushing (rotation), and various actions like mail and disk space events.
/etc/audit/audit.rules lists auditing rules that are loaded on
the service start, restart or reload.
But we will start with using auditctl from the command line. You will need root or sudo to be
able to conduct system-wide auditing. Let’s try an example, which is pretty much what you
will find in the man page. We will configure a watch using auditctl.
Or in text, if you can’t see the image:
auditctl -w /etc/hosts -p war -k hosts-file
What do we have here?
-w flag specifies the file to be watched.
-p sets the permission filter; in other words, what activityyou want to
audit. You have four options, r = read, w = write, x = execute, a = attribute change, which
could be owner, permissions or one of the special file attributes, depending on the
filesystem. -k is a filter, which you can later use to search
through your records in a more meaningful manner. Call it a tag if you will.
Other useful flags
There are many more flags for auditctl. For instance:
-W will remove the watch on a specified file.
-S allows you to trace specific system calls. You can specify
them individually or use all for all system calls. This is somewhat similar to -e flag for
strace, as we’ve seen in the separate tutorial.
Flags -a and -A allow you to add
rules to the end or the beginning or the rules list, respectively. Now, -d will delete a specific rule. -D will
delete all rules.
Adding rules uses a list,action pair, where you first specify
thelist entry, which could be one of the following: task, which is generated when a process
is created, entry, when you enter a system call, exit when you exit a system call, and so
forth. Action can be never and always.
We will see more examples later on. Now, we have a watch in place, let’s edit our /etc/hosts
file and then see what ausearch gives us.
ausearch can be run any time you want, while auditd is running. The output produced is
extremely detailed and should not be read at face value. You may probably want to parse the
information, to grab only whatever interests you.
The basic usage is:
But this will produce an ugly output. You can use the -i flag to make it more reasonable to
read; even so, you will have a lot of data to process.
What do we have here?
type tells us what kind of record we have. For file watch
rules, it will normally be PATH. For system calls, it will be SYSCALL.
msg is the record timestamp. name tells us
the filename for files. You also have file type, permissions, owner, group, device major and
minor numbers, as well as the inode number.
For system calls, it’s a little more complicated. syscall tells
us what system call was, uh, well, called. success tells us if
it worked. You also get the process id (pid) and the parent process id (ppid). There’s a long
set of real and effective user and group ids. You also have the console number.
exe specifies the command that was run. And so forth. Detailed and only
useful for experienced administrators knowing what they need.
This is very similar to going under /proc and burrowing into data for specific processes, but
the thing is, it’s done automatically and faster than a human could possibly log information.
Other useful flags
-k will search by specific keys (tags) only. Otherwise, why did
we use them in the first place? -m will filter only specific
types of records. -sc will match only specific system calls.
-p will only match specific processes.
You can also redirect your output into text editors or files.
There are quite a few examples in the man page. Let me be a monkey and repeat some of them
here, for the sake of education.
To see files opened by a specific user:
auditctl -a exit,always -S open -F auid=510
We will always monitor the exit from the open system call for the audit uid (auid) 510, which
might correspond to some user, like roger. auid is the original ID the user logged in, so if
he/she changes his effective UID while working, it won’t be masked in the log. We want to
monitor the exit from the system call, because if it fails, we will know the exact error.
Another example is:
auditctl -a exit,never -S mount
It says, don’t log any mount system calls.
Here, rather than collecting as much as we can, we suppress output. This could be useful if
you know you might have too many logs or the information is just not interesting. For
example, in the NFS environment, you may not want to do any auditing on the client side and
will instead focus on doing it on the server side. Besides, in this case, there are other
tools you may want to use. For example, pacct, access control via access.conf, traffic
logging, export options for your NFS areas, and more.
One more for ausearch:
ausearch -ts today -k hosts-file -x vi
-ts tells us the timestamp we want to search through our logs; today, only. -k narrows down
our search to the hosts-file key only. -x specifies the exact name of the executable run
against the watched file. In this regard, our example looks for any attempts to open or
modify the file with vi text editor today.
That’s enough, I think.
There are more helper tools that can help you work with auditing. One such tool is
aureport, which you can use to generate summary reports of
collected logs. This is somewhat similar to -c flag in strace.
A handful of my system administration oriented articles.
Some external links:
Finally, man pages:
And that would be all folks!
You have just gained another powerful tool that you can use to make your Linux environment
smarter and better controlled. Alongside an ever-growing repertoire of powerful utilities,
audit will help you build your Linux street credit and make your work easier. audit is very
simple to use, although processing the collected logs can take time and effort. You need to
master all the little flags and understand how the system works before you can delve deeper.
If you have any useful tools you considered using but didn’t dare try, feel free to ping me
and I will testdrive them for you, trying to provide you with the best and most modular
advice while striving for strategic imperatives rather than tactical keyboard hammering.
About the author:
Igor Ljubuncic aka Dedoimedo is the guy behind dedoimedo.com. He makes a living out of his
very hobby – Linux, and holds a bunch of certifications that make a nice pile in the bottom