Category Archives: Security visualization

As networks become ever more complex, securing them becomes more and more difficult. The solution is visualization. Using today’s state-of-the-art data visualization techniques, you can gain a far deeper understanding of what’s happening on your network right now. You can uncover hidden patterns of data, identify emerging vulnerabilities and attacks, and respond decisively with countermeasures that are far more likely to succeed than conventional methods.

Unix Auditd Authentication Events Analysis and Visualisations with ArcSight Logger

Log Management, Security visualization,

With the free ArcSight Logger L750MB, you can in combination with auditd SmartConnector, gather some useful informations in order to have a better overview on your Unix infrastructure Access Management or to respond to some compliances (ex : PCI-DSS, etc.).

In this blog post we will show some examples on how use ArcSight Logger search engine and visualisation capabilities.

Unix Auditd authentication events analysis

We will first analyse Unix auditd authentication events in order to understand what useful informations we can find.

We assume that login under root account is not allowed and that 500 represent the first usable user account.

We will describe here under different SSH authentication auditd events with they results. You can find in bold some important keywords.

  • Successful SSH authentication auditd events
type=USER_AUTH msg=audit(1316335353.969:348): user pid=32370 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=ssh res=success)'

type=USER_LOGIN msg=audit(1316335353.991:355): user pid=32370 uid=0 auid=500 msg='uid=500: exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=/dev/pts/0 res=success)'
  • Failed SSH authentication, for a valid user account, auditd events
type=USER_AUTH msg=audit(1316335438.989:362): user pid=32395 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=ssh res=failed)'

type=USER_LOGIN msg=audit(1316335438.990:363): user pid=32395 uid=0 auid=4294967295 msg='acct="eromang": exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=sshd res=failed)'
  • Failed SSH authentication, for non existing user account, auditd events
type=USER_AUTH msg=audit(1316335506.816:372): user pid=32403 uid=0 auid=4294967295 msg='PAM: authentication acct="?" : exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=ssh res=failed)'

type=USER_LOGIN msg=audit(1316335506.817:373): user pid=32403 uid=0 auid=4294967295 msg='acct="invaliduser": exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=sshd res=failed)'
  • Successful TTY authentication auditd events
type=USER_AUTH msg=audit(1316338747.834:381): user pid=2678 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=success)'

type=USER_LOGIN msg=audit(1316338747.867:437): user pid=2678 uid=0 auid=500 msg='op=login id=500 exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=success)'
  • Failed TTY authentication, for a valid user account, auditd events
type=USER_AUTH msg=audit(1316338880.990:495): user pid=32452 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'

type=USER_LOGIN msg=audit(1316338880.990:496): user pid=32452 uid=0 auid=4294967295 msg='op=login id=500 exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'
  • Failed TTY authentication, for non existing user account, auditd events
type=USER_AUTH msg=audit(1316338916.360:497): user pid=32452 uid=0 auid=4294967295 msg='PAM: authentication acct="?" : exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'

type=USER_LOGIN msg=audit(1316338916.360:498): user pid=32452 uid=0 auid=4294967295 msg='op=login acct="invaliduser" exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'

For USER_LOGIN auditd type events you will have these results :

  • Count of successful and failed authentication attempts (number of lines in the log).
  • Targeted username through auditd “acct” value, but only for failed attempts.
  • Targeted UID for successful and failed attempts through auditd “uid“, “id” and “auid” values.
  • Targeted device address and hostname (box hosting the auditd.log file, or IP/hostname present in SYSLOG log file).
  • Source address and hostname, only for remote authentication, through auditd “hostname” and “addr” values.
  • Associated terminal through “terminal” value.

FOR USER_AUTH type you will have these results :

  • Count of successful and failed authentication attempts (number of lines in the log).
  • Targeted user name through auditd “acct” value, but only valid user accounts.
  • Targeted device address and hostname (box hosting the auditd.log file, or IP/hostname present in SYSLOG log file).
  • Source address and hostname,  only for remote authentication, through auditd “hostname” and “addr” values.
  • Associated terminal through “terminal” value.

In order to provide Access Management analysis, we will focus on USER_AUTH type. With this type we can detect regular authentications and authentication brute force attacks on valid users, with valid user name display, “acct” value.

Unix Auditd authentication events searches in ArcSight Logger

On ArcSight Logger we can now play with searches in order to retrieve and analyse auditd authentication events.

The following filter will provide you all Unix auditd events.

deviceVendor = "Unix" AND deviceProduct = "auditd"

The following filter will provide you all auditd USER_AUTH type events.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"

In order to analyse Access Management activities, we need to understand how the SmartConnector is categorizing the auditd USER_AUTH event and gather the most important fields.

  • baseEventCount : The number of occurrence of the event if aggregation is active on the SmartConnector.
  • deviceAddress : Targeted device address
  • deviceHostName : Targeted device hostname.
  • sourceAddress : The attacker address.
  • sourceHostName : The attacker hostname, if available.
  • sourceUserName : The targeted authentication username.
  • destinationProcessName : Targeted process name. “/usr/sbin/sshd” for SSH or “/bin/login” for TTY console in our cases.
  • deviceCustomString3 : The authentication result. “success” or “failed” in our cases.
  • deviceCustomString6 : For the associated terminal or tty. “ssh” or “ttyx” in our cases.
  • deviceReceiptTime : The event associated timestamp.

In the following scenario, the target was under SSH brute force attack, and the attacker has gain access to the box. We will conduct an analysis, provide you some useful search queries and operators, and try to demonstrate you that using ArcSight Logger for forensics analysis is quiet easy.

This search query is based on the last 24 hours, and we can directly see through the Logger radar that a lot of USER_AUTH requests have been made. In order to only focus on the specific fields, we will execute the following query.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime baseEventCount

“cef” search operator will extracts values for specified fields from matching CEF events. To view only the extracted values, select “User Defined Fieldsets” in the search drill down menu. “cef” search operator will also allow you to use other search operators such as “sort“, “chart“, etc.

To focus on failed authentication attempts, execute the following query.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND deviceCustomString3 = "failed" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString6 deviceReceiptTime  baseEventCount

To focus on successful authentication, execute the following query.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND deviceCustomString3 = "success" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString6 deviceReceiptTime  baseEventCount

Browsing in the events is a quiet boring, ArcSight Logger provide you a query operator how will permit you to create visualisations with your search results.

  • Chart creation to count sourceUserName occurrences

Don’t forget that USER_AUTH auditd type don’t provide the targeted user name for non existing users, so we will only focus on existing sourceUserName.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName

Associated to the result chart, you will also have a result table.

You can also, change the type of chart, by clicking on the upper right corner icon of the result chart frame. The available chart type are column, bar, pie, area, line, stacked column or stacked bar.

  • Chart creation to count Terminal occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by deviceCustomString6

We can see that majority of the authentications are through SSH and that some are through TTY.

  • Chart creation to count sourceUserName and Terminal occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString6

  • Chart creation to count sourceUserName and authentication occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString3

We can see that only user name “eromang” has successful login.

  • Chart creation to count sourceUserName “eromang“, authentication and Terminal occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName = "eromang" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString3 deviceCustomString6

We can discover that user name “eromang” has successful login through “ssh” and “tty” terminals.

  • Chart creation to count sourceUserName “eromang” and successful authentication occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName = "eromang" AND deviceCustomString3 = "success" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString3 deviceCustomString6

  • Char creation to count sourceUserName “eromang”, successful “ssh” authentication and sourceAddress occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName = "eromang" AND deviceCustomString3 = "success" AND deviceCustomString6 = "ssh" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceAddress

We can discover two different source IP addresses.

  • Chart creation to count “ssh” authentications for both sourceAddress occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND (sourceAddress = "192.168.178.25" OR sourceAddress = "192.168.178.21")  AND sourceUserName = "eromang" AND deviceCustomString6 = "ssh"
| cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by deviceCustomString3 sourceAddress

We can discover that sourceAddress “192.168.178.21” has hundreds of failed login and one successful login.

  • Query to find the exact time sourceAddress has breach the target
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND sourceAddress = "192.168.178.21" AND sourceUserName = "eromang" AND deviceCustomString6 = "ssh" AND deviceCustomString3 = "success" | cef deviceReceiptTime

As you can see by this example, ArcSight Logger has some powerful search engine queries and visualisation outputs how will help you to do forensics investigations for your assets.

Honeynet Forensic Challenge 5 Log Mysteries – Analysis – Part 2

Log Management, Security visualization

Following my last post “Part 1 : SSH activities forensics“, I continue to analyse the SSH activities.

Emerging Threats is defining a default “SSH Brute Force” rule, “2001219“, with these triggers :

  • The source IP should be the same
  • The destination port should be 22/tcp
  • The sensor target should be unique (not the ssh server)
  • The thresholds are 5 matching conditions within 2 minutes

Emerging Threats rule don’t care about :

  • If multiples SSH server are in the same $HOME_NET
  • If the targeted user name is unique or not

ArcSight is defining a default “Brute Force Logins”, with these triggers :

  • The source IP should be the same
  • The target IP should be the same
  • The target port should be the same
  • The target username should be the same
  • The thresholds are 5 matching conditions within 2 minutes

We will then consider that 5 matching conditions within 2 minutes by the same source IP as our trigger to detect a SSH brute force attack.

First, we will check the stats for each source IPs how have fail to access the box and never succeeded (third Google gadget).

All the failed SSH connexions uper or equal to 5 in a timeframe of 120 seconds will be considered as this kind of attack.

The detailed stats are avaible in this Google Document under the “SSH – 5” worksheet.

On 25 source IPs, how have fail to access the box and never succeeded, 19 would be detected by ET 2001219 rule.

4 of the 6 source IPs, non detected as a brute force attack, with a number of connexion uper than 20 are interesting, it is a real non detected brute force attack.

On ArcSight, with the default “Brute Force Logins” correlation rule, only 13 on 25 source IPs would be detected as a brute force attack.

Now we will do the same analysis for each source IPs how have fail to access the box and finaly succeeded (second Google gadget).

All the failed SSH connexions uper or equal to 5 in a timeframe of 120 seconds will be considered as this kind of attack.

The detailed stats are avaible in this Google Document under the “SSH – 6” worksheet.

On 7 source IPs, how have fail to access the box and never succeeded, 6 would be detected, by ET 2001219 rule.

On ArcSight, with the default “Brute Force Logins” correlation rule, only 4 on 7 source IPs would be detected as a brute force attack.

More the “Brute Force Attacks” attempts are distributed in time, more patient you are, more you switch between a few targeted user accounts, less you will be discovered. ArcSight and Emerging Threats default rules will detect nothing.

Honeynet Forensic Challenge 5 Log Mysteries – Analysis – Part 1

Log Management, Security visualization

The Honeynet Project has launch a new forensic challenge, for year 2010, called “Challenge 5 – Log Mysteries“. This challenge is provided by Raffael Marty, Anton Chuvakin and Sebastien Tricaud and take the participant into the world of virtual systems and confusing log data. In this challenge, the participant should figure out what happened to a virtual server using the logs from a possible compromised server.

I will maybe officially participate to this challenge, but I will post on my blog all analysis I done around it. If you have any suggestions or corrections don’t hesitate to post a comment on the blog, or contact me by twitter.

Part 1 : SSH activities forensics

All SSH activities are stored into the “auth.log” file with “sshd” string a primary key for all searches.

  • Gathering and visualizing all SSH failed access attempt

Failed SSH access on the box are identified by string “Failed password for invalid user xxx” or “Failed password for xxx“, where xxx is representing the targeted user account. The source IP addresses are also provided by the logs.

To get a list of all failed access source IPs and the number of failed access occurrences by source IPs, just run the following commands. We will mark the failed access as “FAILED” into a CSV “ssh_failed_sips.csv” file for further usages.
cat auth.log | grep sshd | grep "Failed password for invalid user" | awk '{print$13",FAILED"}' > ssh_failed_sips.csv
cat auth.log | grep sshd | grep "Failed password for" | grep -vw 'invalid' | awk '{print$11",FAILED"}' >> ssh_failed_sips.csv

To create a representation of all these failed access, we will use Afterglow in two node mode. The source IPs will be colored in “lightskyblue1“, the event (here FAILED) will be colored in “red” color and represented as a “box“.

cat ssh_failed_sips.csv | afterglow.pl -c color.properties -a -d -e 3 -t | neato -Tpng -o ssh_failed_sips.png

The Afterglow “-d” option will display the node count.

The Afterglow visualization of the SSH failed access attempt is available here.

With this visualization we can see, by the node count, that some “SSH Brute force” attempts were done on the server from some source IPs.

  • Gathering and visualizing all SSH success access

Accepted SSH access on the box are identified by string “Accepted password for xxx“, where xxx is representing the targeted user account. The source IP addresses are also provided by the logs.

To get a list of all success source IPs and the number of success access occurrences by source IPs, just run the following commands. We will mark the success access as “SUCCESS” into a CSV file “ssh_success_sips.csv” for further usages.

cat auth.log | grep sshd | grep "Accepted password for" | awk '{print$11",SUCCESS"}' > ssh_success_sips.csv

To create a representation of all these success access, we will use Afterglow in two node mode. The source IPs will be colored in “lightskyblue1“, the event (here SUCCESS) will be colored in “green” color and represented as a “box“.

cat ssh_success_sips.csv | afterglow.pl -c color.properties -a -d -e 3 -t | neato -Tpng -o ssh_success_sips.png

The Afterglow “-d” option will display the node count.

The Afterglow visualization of the SSH success access attempt is available here.

We can see a large amount of successful access on the box from different source IPs.

  • Merging successful and failed SSH access

Now we have all successful and failed SSH access, we will merge all these datas into another CSV file “ssh_failed_success_sips.csv” to have a global view.

cat ssh_failed_sips.csv > ssh_failed_success_sips.csv
cat ssh_success_sips.csv >> ssh_failed_success_sips.csv

To create a representation of all these access, we will use Afterglow in two node mode. The source IPs will be colored in “lightskyblue1“, the success event (SUCCESS) will be colored in green, the failed event (FAILED) will be colored in red, and represented as a “box“.

cat ssh_failed_success_sips.csv | afterglow.pl -c color.properties -a -d -e 3 -t | neato -Tpng -o ssh_failed_success_sips.png

The Afterglow visualization of the SSH all access attempt is available here.

Here under a Google gadget for all SSH success or failed access with count by result.

We can see that some source IPs have both failed and success attempts, some of them with “brute force attacks”. Other source IPs have only success access to the box. Now we will create a CSV file “ssh_success_unique_sips.csv” with only the source IPs how have successfully access the box. This new CSV file will be used to create another CSV file “ssh_sips_stats.csv” representing all SSH success and failed access attempt, based on the unique source IPs how have successful access the box.

for i in `cat ssh_success_unique_sips.csv`; do success=`cat ssh_failed_sips.csv | grep $i | wc -l`; failed=`cat ssh_success_sips.csv | grep $i | wc -l`; echo $i,"FAILED",$success >> ssh_sips_stats.csv; echo $i,"SUCCESS",$failed >> ssh_sips_stats.csv; done

All the results have been imported into a Google gadget available here under.

With the CSV file “ssh_failed_success_sips.csv“ we will create a new CSV file “all_ssh_sips.csv” containing all source IPs how have try or access the box.

cat ssh_failed_success_sips.csv | awk -F "," '{print$1}' | sort -u > all_ssh_sips.csv

Based on this new CSV file “all_ssh_sips.csv” and the existing CSV file “ssh_success_unique_sips.csv” we will now create a dedicated CSV file “ssh_failed_sips.csv” for source IPs how have fail to access the box and never succeeded.

comm -13 ssh_success_unique_sips.csv all_ssh_sips.csv > ssh_failed_sips.csv

By executing this script, you will have stats for each source IPs how have fail to access the box and never succeeded. This script will create a CSV file called “ssh_failed_sips_count.csv“.

./ssh_failed_sips_count.sh

Here under you can get a Google gadget with all these datas.

  • Detailed statistics for success source IPs

I developed a bash script to get all detailed statistics for the source IPs how have successfully SSH access to the box. This script “ssh_sips_detailled_stats.sh” give you detailled results “ssh_sips_detailled_stats.txt” (First seen, last seen, number of events, number of different logins, first successful SSH connexion, last successful SSH connexion, timeline of all successful SSH connexions, number of failed connexions). Another datas could be provided by executing the following command how will create a new CSV file “ssh_timeline.csv” with all the success SSH access timeline. Here under a Google gadget with all the timeline results.

More in the next post…