Following my last post “Part 1 : SSH activities forensics“, I continue to analyse the SSH activities.

Emerging Threats is defining a default “SSH Brute Force” rule, “2001219“, with these triggers :

• The source IP should be the same
• The destination port should be 22/tcp
• The sensor target should be unique (not the ssh server)
• The thresholds are 5 matching conditions within 2 minutes

Emerging Threats rule don’t care about :

• If multiples SSH server are in the same $HOME_NET
• If the targeted user name is unique or not

ArcSight is defining a default “Brute Force Logins”, with these triggers :

• The source IP should be the same
• The target IP should be the same
• The target port should be the same
• The target username should be the same
• The thresholds are 5 matching conditions within 2 minutes

We will then consider that 5 matching conditions within 2 minutes by the same source IP as our trigger to detect a SSH brute force attack.

First, we will check the stats for each source IPs how have fail to access the box and never succeeded (third Google gadget).

All the failed SSH connexions uper or equal to 5 in a timeframe of 120 seconds will be considered as this kind of attack.

The detailed stats are avaible in this Google Document under the “SSH – 5” worksheet.

On 25 source IPs, how have fail to access the box and never succeeded, 19 would be detected by ET 2001219 rule.

4 of the 6 source IPs, non detected as a brute force attack, with a number of connexion uper than 20 are interesting, it is a real non detected brute force attack.

On ArcSight, with the default “Brute Force Logins” correlation rule, only 13 on 25 source IPs would be detected as a brute force attack.

Now we will do the same analysis for each source IPs how have fail to access the box and finaly succeeded (second Google gadget).

All the failed SSH connexions uper or equal to 5 in a timeframe of 120 seconds will be considered as this kind of attack.

The detailed stats are avaible in this Google Document under the “SSH – 6” worksheet.

On 7 source IPs, how have fail to access the box and never succeeded, 6 would be detected, by ET 2001219 rule.

On ArcSight, with the default “Brute Force Logins” correlation rule, only 4 on 7 source IPs would be detected as a brute force attack.

More the “Brute Force Attacks” attempts are distributed in time, more patient you are, more you switch between a few targeted user accounts, less you will be discovered. ArcSight and Emerging Threats default rules will detect nothing.

The Honeynet Project has launch a new forensic challenge, for year 2010, called “Challenge 5 – Log Mysteries“. This challenge is provided by Raffael Marty, Anton Chuvakin and Sebastien Tricaud and take the participant into the world of virtual systems and confusing log data. In this challenge, the participant should figure out what happened to a virtual server using the logs from a possible compromised server.

I will maybe officially participate to this challenge, but I will post on my blog all analysis I done around it. If you have any suggestions or corrections don’t hesitate to post a comment on the blog, or contact me by twitter.

Part 1 : SSH activities forensics

All SSH activities are stored into the “auth.log” file with “sshd” string a primary key for all searches.

Failed SSH access on the box are identified by string “Failed password for invalid user xxx” or “Failed password for xxx“, where xxx is representing the targeted user account. The source IP addresses are also provided by the logs.

cat auth.log | grep sshd | grep "Failed password for invalid user" | awk '{print$13",FAILED"}' > ssh_failed_sips.csv
cat auth.log | grep sshd | grep "Failed password for" | grep -vw 'invalid' | awk '{print$11",FAILED"}' >> ssh_failed_sips.csv

To create a representation of all these failed access, we will use Afterglow in two node mode. The source IPs will be colored in “lightskyblue1“, the event (here FAILED) will be colored in “red” color and represented as a “box“.

cat ssh_failed_sips.csv | afterglow.pl -c color.properties -a -d -e 3 -t | neato -Tpng -o ssh_failed_sips.png

The Afterglow “-d” option will display the node count.

The Afterglow visualization of the SSH failed access attempt is available here.

With this visualization we can see, by the node count, that some “SSH Brute force” attempts were done on the server from some source IPs.

• Gathering and visualizing all SSH success access

Accepted SSH access on the box are identified by string “Accepted password for xxx“, where xxx is representing the targeted user account. The source IP addresses are also provided by the logs.

To get a list of all success source IPs and the number of success access occurrences by source IPs, just run the following commands. We will mark the success access as “SUCCESS” into a CSV file “ssh_success_sips.csv” for further usages.

cat auth.log | grep sshd | grep "Accepted password for" | awk '{print$11",SUCCESS"}' > ssh_success_sips.csv

To create a representation of all these success access, we will use Afterglow in two node mode. The source IPs will be colored in “lightskyblue1“, the event (here SUCCESS) will be colored in “green” color and represented as a “box“.

cat ssh_success_sips.csv | afterglow.pl -c color.properties -a -d -e 3 -t | neato -Tpng -o ssh_success_sips.png

The Afterglow “-d” option will display the node count.

The Afterglow visualization of the SSH success access attempt is available here.

We can see a large amount of successful access on the box from different source IPs.

• Merging successful and failed SSH access

Now we have all successful and failed SSH access, we will merge all these datas into another CSV file “ssh_failed_success_sips.csv” to have a global view.

cat ssh_failed_sips.csv > ssh_failed_success_sips.csv
cat ssh_success_sips.csv >> ssh_failed_success_sips.csv

To create a representation of all these access, we will use Afterglow in two node mode. The source IPs will be colored in “lightskyblue1“, the success event (SUCCESS) will be colored in green, the failed event (FAILED) will be colored in red, and represented as a “box“.

cat ssh_failed_success_sips.csv | afterglow.pl -c color.properties -a -d -e 3 -t | neato -Tpng -o ssh_failed_success_sips.png

The Afterglow visualization of the SSH all access attempt is available here.

Here under a Google gadget for all SSH success or failed access with count by result.

We can see that some source IPs have both failed and success attempts, some of them with “brute force attacks”. Other source IPs have only success access to the box. Now we will create a CSV file “ssh_success_unique_sips.csv” with only the source IPs how have successfully access the box. This new CSV file will be used to create another CSV file “ssh_sips_stats.csv” representing all SSH success and failed access attempt, based on the unique source IPs how have successful access the box.

for i in `cat ssh_success_unique_sips.csv`; do success=`cat ssh_failed_sips.csv | grep $i | wc -l`; failed=`cat ssh_success_sips.csv | grep $i | wc -l`; echo $i,"FAILED",$success >> ssh_sips_stats.csv; echo $i,"SUCCESS",$failed >> ssh_sips_stats.csv; done

All the results have been imported into a Google gadget available here under.

With the CSV file “ssh_failed_success_sips.csv“ we will create a new CSV file “all_ssh_sips.csv” containing all source IPs how have try or access the box.

cat ssh_failed_success_sips.csv | awk -F "," '{print$1}' | sort -u > all_ssh_sips.csv

Based on this new CSV file “all_ssh_sips.csv” and the existing CSV file “ssh_success_unique_sips.csv” we will now create a dedicated CSV file “ssh_failed_sips.csv” for source IPs how have fail to access the box and never succeeded.

comm -13 ssh_success_unique_sips.csv all_ssh_sips.csv > ssh_failed_sips.csv

By executing this script, you will have stats for each source IPs how have fail to access the box and never succeeded. This script will create a CSV file called “ssh_failed_sips_count.csv“.

./ssh_failed_sips_count.sh

Here under you can get a Google gadget with all these datas.

• Detailed statistics for success source IPs

I developed a bash script to get all detailed statistics for the source IPs how have successfully SSH access to the box. This script “ssh_sips_detailled_stats.sh” give you detailled results “ssh_sips_detailled_stats.txt” (First seen, last seen, number of events, number of different logins, first successful SSH connexion, last successful SSH connexion, timeline of all successful SSH connexions, number of failed connexions). Another datas could be provided by executing the following command how will create a new CSV file “ssh_timeline.csv” with all the success SSH access timeline. Here under a Google gadget with all the timeline results.

More in the next post…

On demand of the Honeynet Team, and to be fair, I will switch off all Blog posts related to the Log Mysteries Challenge. All the analysis will be posted at the end of the challenge, the 30 September 2010.