Category Archives: Log Management

Log management comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc.). Log management covers log collection, centralized aggregation, long-term retention, log analysis (in real-time and in bulk after storage) as well as log search and reporting. Log management is driven by reasons of security, system and network operations (such as system or network administration) and regulatory compliance. Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs). Wikipedia definition.

Unix Auditd Authentication Events Analysis and Visualisations with ArcSight Logger

Log Management, Security visualization,

With the free ArcSight Logger L750MB, you can in combination with auditd SmartConnector, gather some useful informations in order to have a better overview on your Unix infrastructure Access Management or to respond to some compliances (ex : PCI-DSS, etc.).

In this blog post we will show some examples on how use ArcSight Logger search engine and visualisation capabilities.

Unix Auditd authentication events analysis

We will first analyse Unix auditd authentication events in order to understand what useful informations we can find.

We assume that login under root account is not allowed and that 500 represent the first usable user account.

We will describe here under different SSH authentication auditd events with they results. You can find in bold some important keywords.

  • Successful SSH authentication auditd events
type=USER_AUTH msg=audit(1316335353.969:348): user pid=32370 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=ssh res=success)'

type=USER_LOGIN msg=audit(1316335353.991:355): user pid=32370 uid=0 auid=500 msg='uid=500: exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=/dev/pts/0 res=success)'
  • Failed SSH authentication, for a valid user account, auditd events
type=USER_AUTH msg=audit(1316335438.989:362): user pid=32395 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=ssh res=failed)'

type=USER_LOGIN msg=audit(1316335438.990:363): user pid=32395 uid=0 auid=4294967295 msg='acct="eromang": exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=sshd res=failed)'
  • Failed SSH authentication, for non existing user account, auditd events
type=USER_AUTH msg=audit(1316335506.816:372): user pid=32403 uid=0 auid=4294967295 msg='PAM: authentication acct="?" : exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=ssh res=failed)'

type=USER_LOGIN msg=audit(1316335506.817:373): user pid=32403 uid=0 auid=4294967295 msg='acct="invaliduser": exe="/usr/sbin/sshd" (hostname=macbook.zataz.loc, addr=192.168.178.25, terminal=sshd res=failed)'
  • Successful TTY authentication auditd events
type=USER_AUTH msg=audit(1316338747.834:381): user pid=2678 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=success)'

type=USER_LOGIN msg=audit(1316338747.867:437): user pid=2678 uid=0 auid=500 msg='op=login id=500 exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=success)'
  • Failed TTY authentication, for a valid user account, auditd events
type=USER_AUTH msg=audit(1316338880.990:495): user pid=32452 uid=0 auid=4294967295 msg='PAM: authentication acct="eromang" : exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'

type=USER_LOGIN msg=audit(1316338880.990:496): user pid=32452 uid=0 auid=4294967295 msg='op=login id=500 exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'
  • Failed TTY authentication, for non existing user account, auditd events
type=USER_AUTH msg=audit(1316338916.360:497): user pid=32452 uid=0 auid=4294967295 msg='PAM: authentication acct="?" : exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'

type=USER_LOGIN msg=audit(1316338916.360:498): user pid=32452 uid=0 auid=4294967295 msg='op=login acct="invaliduser" exe="/bin/login" (hostname=?, addr=?, terminal=tty1 res=failed)'

For USER_LOGIN auditd type events you will have these results :

  • Count of successful and failed authentication attempts (number of lines in the log).
  • Targeted username through auditd “acct” value, but only for failed attempts.
  • Targeted UID for successful and failed attempts through auditd “uid“, “id” and “auid” values.
  • Targeted device address and hostname (box hosting the auditd.log file, or IP/hostname present in SYSLOG log file).
  • Source address and hostname, only for remote authentication, through auditd “hostname” and “addr” values.
  • Associated terminal through “terminal” value.

FOR USER_AUTH type you will have these results :

  • Count of successful and failed authentication attempts (number of lines in the log).
  • Targeted user name through auditd “acct” value, but only valid user accounts.
  • Targeted device address and hostname (box hosting the auditd.log file, or IP/hostname present in SYSLOG log file).
  • Source address and hostname,  only for remote authentication, through auditd “hostname” and “addr” values.
  • Associated terminal through “terminal” value.

In order to provide Access Management analysis, we will focus on USER_AUTH type. With this type we can detect regular authentications and authentication brute force attacks on valid users, with valid user name display, “acct” value.

Unix Auditd authentication events searches in ArcSight Logger

On ArcSight Logger we can now play with searches in order to retrieve and analyse auditd authentication events.

The following filter will provide you all Unix auditd events.

deviceVendor = "Unix" AND deviceProduct = "auditd"

The following filter will provide you all auditd USER_AUTH type events.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"

In order to analyse Access Management activities, we need to understand how the SmartConnector is categorizing the auditd USER_AUTH event and gather the most important fields.

  • baseEventCount : The number of occurrence of the event if aggregation is active on the SmartConnector.
  • deviceAddress : Targeted device address
  • deviceHostName : Targeted device hostname.
  • sourceAddress : The attacker address.
  • sourceHostName : The attacker hostname, if available.
  • sourceUserName : The targeted authentication username.
  • destinationProcessName : Targeted process name. “/usr/sbin/sshd” for SSH or “/bin/login” for TTY console in our cases.
  • deviceCustomString3 : The authentication result. “success” or “failed” in our cases.
  • deviceCustomString6 : For the associated terminal or tty. “ssh” or “ttyx” in our cases.
  • deviceReceiptTime : The event associated timestamp.

In the following scenario, the target was under SSH brute force attack, and the attacker has gain access to the box. We will conduct an analysis, provide you some useful search queries and operators, and try to demonstrate you that using ArcSight Logger for forensics analysis is quiet easy.

This search query is based on the last 24 hours, and we can directly see through the Logger radar that a lot of USER_AUTH requests have been made. In order to only focus on the specific fields, we will execute the following query.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime baseEventCount

“cef” search operator will extracts values for specified fields from matching CEF events. To view only the extracted values, select “User Defined Fieldsets” in the search drill down menu. “cef” search operator will also allow you to use other search operators such as “sort“, “chart“, etc.

To focus on failed authentication attempts, execute the following query.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND deviceCustomString3 = "failed" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString6 deviceReceiptTime  baseEventCount

To focus on successful authentication, execute the following query.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND deviceCustomString3 = "success" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString6 deviceReceiptTime  baseEventCount

Browsing in the events is a quiet boring, ArcSight Logger provide you a query operator how will permit you to create visualisations with your search results.

  • Chart creation to count sourceUserName occurrences

Don’t forget that USER_AUTH auditd type don’t provide the targeted user name for non existing users, so we will only focus on existing sourceUserName.

deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName

Associated to the result chart, you will also have a result table.

You can also, change the type of chart, by clicking on the upper right corner icon of the result chart frame. The available chart type are column, bar, pie, area, line, stacked column or stacked bar.

  • Chart creation to count Terminal occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by deviceCustomString6

We can see that majority of the authentications are through SSH and that some are through TTY.

  • Chart creation to count sourceUserName and Terminal occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString6

  • Chart creation to count sourceUserName and authentication occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName IS NOT NULL | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString3

We can see that only user name “eromang” has successful login.

  • Chart creation to count sourceUserName “eromang“, authentication and Terminal occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName = "eromang" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString3 deviceCustomString6

We can discover that user name “eromang” has successful login through “ssh” and “tty” terminals.

  • Chart creation to count sourceUserName “eromang” and successful authentication occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName = "eromang" AND deviceCustomString3 = "success" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceUserName deviceCustomString3 deviceCustomString6

  • Char creation to count sourceUserName “eromang”, successful “ssh” authentication and sourceAddress occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH" AND sourceUserName = "eromang" AND deviceCustomString3 = "success" AND deviceCustomString6 = "ssh" | cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by sourceAddress

We can discover two different source IP addresses.

  • Chart creation to count “ssh” authentications for both sourceAddress occurences
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND (sourceAddress = "192.168.178.25" OR sourceAddress = "192.168.178.21")  AND sourceUserName = "eromang" AND deviceCustomString6 = "ssh"
| cef deviceAddress deviceHostName sourceAddress sourceHostName sourceUserName destinationProcessName deviceCustomString3 deviceCustomString6 deviceReceiptTime  baseEventCount | chart sum(baseEventCount) AS Total by deviceCustomString3 sourceAddress

We can discover that sourceAddress “192.168.178.21” has hundreds of failed login and one successful login.

  • Query to find the exact time sourceAddress has breach the target
deviceVendor = "Unix" AND deviceProduct = "auditd" AND deviceEventCategory = "USER_AUTH"  AND sourceAddress = "192.168.178.21" AND sourceUserName = "eromang" AND deviceCustomString6 = "ssh" AND deviceCustomString3 = "success" | cef deviceReceiptTime

As you can see by this example, ArcSight Logger has some powerful search engine queries and visualisation outputs how will help you to do forensics investigations for your assets.

Cisco Smart Business Architecture (SBA) guides for SIEM solutions integration

Event Management, Log Management, , , , , ,

Cisco provide some useful Smart Business Architecture (SBA) guides for SIEM solutions integration how will helps you to design and deploy best practices that include Cisco switching, routing, security and wireless technologies.

Actually the SBA guides are covering the following solutions :

  • SBA guide how provides a general overview of SIEM technology, as well as best practices, use cases, and deployment considerations for using a SIEM with Cisco infrastructure (click here to read). Cisco products logging retrieval methods,
  • SBA guide for ArcSight SIEM plateform (ESM, Logger, Express, SmartConnectors and Content Pack) integration (click here to read).
  • SBA guide for Loglogic MX Series SIEM product integration (click here to read).
  • SBA guide for netForensics nFX Cinxi One SIEM product integration (click here to read).
  • SBA guide for RSA enVision SIEM product integration (click here to read).
  • SBA guide for Splunk security management solution (click here to read).

ArcSight Logger File Receiver Configuration

Log Management,

ArcSight Logger propose different kind of receivers :

  • UDP receiver for UDP messages, such as SYSLOG.
  • TCP receiver for TCP messages, such as SYSLOG how can also be sent with TCP.
  • CEF UDP receiver for CEF (Common Event Format) messages sent through UDP.
  • CEF TCP receiver for CEF (Common Event Format) messages sent through TCP.
  • SmartMessage receiver for encrypted SmartMessage messages sent by SmartConnectors.
  • File Transfer to read remote logs using scp, sftp or ftp.
  • File Receiver to read logs from a local or remote file system such as NFS, CIFS or SAN.

In my previous blog posts, we configured SmartConnectors to send they’re messages to the Logger SmartMessage receiver. The SmartMessage reception method is the most used in an typical ArcSight Log Management infrastructure. With SmartConnectors and SmartMessage receiver usage you can benefit of :

  • SmartConnector normalization, categorization, aggregation, batching and filtering.
  • SmartMessage encrypted transmission of messages.

But it could happen that you need to collect events without SmartConnector or FlexConnector. For examples, maybe you have an existing network file system (NFS) to centralize all your Apache HTTP Server logs files, or maybe you are not allowed to install SmartConnector on a system and the policy doesn’t allow to send messages through UDP. File Receiver Logger receivers will provide you alternatives in order to still collect the events.

File Receiver Configuration

ArcSight Logger File Receiver allow you to read files from a network file system (NFS) CIFS, or storage area network (SAN). But the free ArcSight Logger L750MB only allow you File Receiver through a local share. So the ArcSight Logger L750MB processes, how are running under “arcsight” user and group should have the permissions to read and/or write into this share.

In order to setup a File Receiver for ArcSight Logger L750MB, first create a “filereceiver01” folder in “$ARCSIGHT_HOME“. Or you can mount a NFS, CIFS or SAN on the Logger filesystem.

Then log into the Logger Web interface and go into “Configuration -> Event Inpout/Output“. Click on the “Add” button, give a name to your File Receiver (for example: FileReceiver01), select “File Receiver” in the pull-down list and click on the “Next” button.

To complete the setup specify the following information’s :

  • RFS Names : On a L750MB Logger you can only select “LOCAL“, but on a appliance Logger you can select the previously declared NFS, CIFS or SAN share.
  • Folder : Specify the folder name, in our example “/home/arcsight/filereceiver01“.
  • Source Type : Select from the pull-down list your log file types. “Microsoft DHCP Log“, “Juniper Steel-Belted Radius“, “Apache HTTP Server Error“, “Apache HTTP Server Access“, “IBM DB2 Audit” or “Other“. The “Other” choice will allow you to import all kind of logs.
  • Wildcard (regex) : Allow you through regular expressions to describe the log file to read. “.*” mean all files.
  • Mode : Select from the pull-down list. With “Persist” mode, the Logger will remember which files have been processed and only processes them once. With “Rename” mode will rename the log file once it has been processed. With “Delete” mode, the file is deleted once it has been processed.
  • Rename extension : If you have select the “Rename” mode, specify the suffix to append to the log files. By default “.done“.
  • Character encoding : Select the log file character encoding.
  • Delay after seen : Specify the number of seconds to wait after a source file is first seen until it is processed. This allows the entire file to be copied before processing begins. By default “10” seconds.
  • Date/time locale : Select from the pull-down list your locale.
  • Date/time zone : Select from the pull-down list your time zone, if no time zone is specified into the log file. By default, the time zone is the Logger time zone.
  • Date/time loc. regex : Allow you through regular expressions to describe which characters represent the timestamps in the log file. By default no timestamp in the log file.
  • Date/time format : If timestamps are present in the log file, specify the format of the timestamps through format specifiers. By default no timestamps in the log file.
  • Event start (regex) : If you’re log files are multi-line, you can specify a regular expression how describe the start of a new event in the log file. By default, each line in the log file is considered as a single event.

Then save your configuration. You will have to active the File Receiver by clicking on the right button.

Directly after activating the File Receiver a Device will be created into “Configuration -> Devices“. In order to search easily on the File Receiver events, I recommend you to create a dedicated Device Group (ex : FileReceiver), from “Configuration -> Devices -> Device Groups“, and to associated the newly created File Receiver with this new Device Group.

Also I recommend you, to dedicate a Storage Group (ex : File Receivers) to all events coming from the File Receivers, from “Configuration -> Storage -> Storage Groups“. You can rename on existing Storage Group and adapt the associated retention period and maximum size.

Associate the created Device Group with the renamed Storage Group by Storage Rules, from “Configuration -> Storage -> Storage Rules).

Finally you have to restart the File Receiver by clicking twice on the right button. If you don’t restart the receiver, the Device and Storage Groups associated with the receiver are not active.

Retrieving events from File Receiver

If you have create dedicated Device Groups and/or Storage Groups for both File Receiver, you will be able to easily find the associated events through the Logger search engine.

For searches on Device Groups, just type the following command, where “FileReceiver” is you Device Group.

For searches on Storage Groups, just type the following command, where “File Receivers” is your Storage Group.

You can also mix all these search parameters.

Receivers Debugging

All activities related to the Logger Receivers are located in the “/home/arcsight/current/arcsight/logger/logs/logger_receiver.log” log file.

“receiver_https” is corresponding to the SmartMessage Receiver, and “rfs_file_receiver” to the newly added File Receiver.

“Eps(SLC)” is corresponding to the current EPS rate, “Total Events” is the total number of events the receiver has processed since the last restart, “Eps(max)” the maximum EPS rate the receiver has processed since the last restart, “Bps(SLC)” the bytes per second the receiver has processed.

Why And Howto Calculate Your Events Log Size

Event Management, Log Management, ,

If you are projecting to start a Log or Event Management project, you will surely need to know your Normal Event log size (NE). These Normal Event log size (NE) value, combinated with the your Normal Events per second (NE) value and with your storage retention policy will help you to design in order to estimate your storage requirements.

Never forget that Log Management storage requirements are not the same for Event Management. Most of time Log Management storage requirements are higher than for Event Management. For example for Log Management, PCI-DSS v2.0 Req. 10.7 require 1 year retention :

10.7 Retain audit trail history for at least one year, with a minimum of three months immediately available for analysis (for example, online, archived, or restorable from back-up).

But in order to compensate PCI-DSS v2.0 Req. 10.6, you will maybe do Event Management with a SIEM (like ArcSight ESM, RSA enVision, QRadar SIEM, etc.).

10.6 Review logs for all system components at least daily. Log reviews must include those servers that perform security functions like intrusion-detection system (IDS) and authentication, authorization, and accounting protocol (AAA) servers (for
example, RADIUS). Note: Log harvesting, parsing, and alerting tools may be used to meet compliance with Requirement 10.6

You don’t need a SIEM to do Log Management, but you also don’t need to store 1 year of your logs on your SIEM solution. Long term retention, long term reporting, “raw” events forensics are mostly done on a Log Management infrastructure (like ArcSight Logger, QRadar Log Manager, Novell Sentinel Log Manager, etc.). Storage retention for your Event Management infrastructure will depend mostly on your correlation rules, your acknowledge time on a correlated event, the number of security analysts present in your SOC, etc.

Don’t imagine that a magic formula exist to define your events log size, some tools could help you, but you need to analyze your logs in order to have your Normal Event log size.  First of all you have to define your Log and/or Event Management scope, this scope could first be driven by regulations or compliances, but don’t forget that regulations or compliances are not Security. Also each technologies have different log sizes, an Apache HTTPD log will not have the same size than a SSHD log, and an Apache HTTPD log from server A will surely not have the same size than an Apache HTTPD log from server B.

xxx.xxx.xxx.xxx - - [25/Aug/2011:04:23:47 +0200] "GET /feed/ HTTP/1.1" 304 - "-" "Apple-PubSub/65.28"

This log from Apache HTTPD server A has a size of 102 bytes.

xxx.xxx.xxx.xxx - - [25/Aug/2011:04:15:08 +0200] "GET /wp-content/themes/mystique/css/style-green.css?ver=3.0.7 HTTP/1.1" 200 1326 "http://eromang.zataz.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.20) Gecko/20110803 Firefox/3.6.20 ( .NET CLR 3.5.30729)"

This log from Apache HTTPD server B has a size of 274 bytes.

Also, depending the Log or Event Management infrastructure product, you need to consider event generated by intrinsically mechanism. For example, in order to search in your events most of products are creating indexes, these indexes are representing an average of twice the time of the size of the event. Also another intrinsically mechanism is that these products are also monitoring themselves, regularly executing tasks, do some statistics for dashboards or reports.

I have develop a bash script how will permit you to analyze all your archived logs and gather the following informations:

  • For each archived files, the total number of events, the total uncompressed size of the events, the Normal Event log size.
  • The total events for all archived files.
  • The total uncompressed size of all events in all archived files.
  • The grant total Normal Event log size.
  • The average event number per archived files.
  • The average bytes per archived file.

You can download this script by clicking on this link. A reminder, the provided Normal Events per second value, is not your real EPS rate, just check my previous blogpost regarding on “Why and howto calculate your Events Per Second“.