Enjoy your IDS (Part 1)

Although one would guess that you install an IDS (or more) within your infrastructure to get alerts and react against them, that is usually far from reality. Many of those systems, even quite a few in big organizations, produce logs that are only reviewed when a serious incident happens. I know that is hard to imagine, but let me guess that it can happen to you too.

Unfortunately the amount of alerts generated by an IDS under normal conditions is normally huge, especially when it hasn't been tuned up for the environment that it lives in. In those cases, reading the alerts one after the other is not an option for the average security professional that must dedicate his time to many other things. As computer geeks we believe that computers can do this kind of tasks better than us. So scripting should be your friend for these cases.

In today's post I will not only provide you with my opinion on how to do things, but I will also do some extra work ( :-) ) and contribute with one simple script to extract some info from your alerts file. So let me first define what the script can do for you.

If you don't read your alerts very often, chances are that you have to check a lot of false positives or, at least, not very relevant alerts. So it would be nice if you could separate your data into groups of different alerts and decide if they are relevant to you, maybe for later investigation, or can be safely ignored.

Let's assume that:

• your IDS is snort,


• the alerts are generated in the default format. One line per alert with Timestamp interface, alert ID, alert message, priority, protocol, source IP address, source port, destination IP address and destination port)

Your first move could then be starting by grouping the alerts by the alert message (i.e. the supposed attack that has been detected). In each of the groups resultant of that classification you can have alerts that can be safely ignored together with others that must be treated as an incident and the difference could be either the source IP address or the destination IP address (maybe, even the time stamp). So what a useful tool could do is classify the alerts into groups using one of the fields, showing the number of alerts per group. To be able to go deeper in your investigation you also want the tool to be able to fix that field to the value of the group that you want to investigate further (i.e. alert equals portscan) and show the data grouped by another field (for example, source IP address). If you use the criteria that I have just mentioned and you find out that you have several portscanning attempts from one internal address, you can investigate further if:

• This system is being used by some evil hacker.


• The system is compromised and some kind of malware is running in it.

This is what this program does. To be able to get the groups of alerts you would use:
alertgrp.py /var/log/snort/alert alert
Then, after the portscan alerts catching your eye, you can try to identify the source IP address by using:
alertgrp.py /var/log/snort/alert srcip alert='.*portscan.*'
You can verify which IP addresses were scanned by the allegedly guilty system using:
alertgrp.py /var/log/snort/alert dstip alert='.*portscan.*',srcip=10.0.0.1

I hope that these examples give you some glimpse of the way the tool can be used. Anyhow, the source is there and you can read it and modify it at your will.

I hope you agree with me that the language that you use to write this script is not very relevant as long as it is able to work with regular expressions and you are familiar enough with it to make it do what you want (Please make my day with a programming language religious war ;-P). In this case I chose Python (instead of Perl or shellscript + awk or grep) because I want to use this script for my next post an integrate this results with a visualization tool to enhance the experience and get more information from a glance to the data. More on that in the best blog near you really soon now.

Please, keep in mind that I am not claiming to by a Python guru (which in fact I am not), so the script might (and it probably will) contain errors. If you find any, let me know and I'll do my best to fix it and update the version. However, using memory to store all the field values instead of just counting them when reading is intentional and you will see why in my next post.

Hope you like to put some regular expressions on your life!