Category Archives: Log Management

Log management comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc.). Log management covers log collection, centralized aggregation, long-term retention, log analysis (in real-time and in bulk after storage) as well as log search and reporting. Log management is driven by reasons of security, system and network operations (such as system or network administration) and regulatory compliance. Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs). Wikipedia definition.

ArcSight SmartConnector Custom Zones Mapping

Once you have install and configure your SYSLOG ArcSight SmartConnector to communicate with your free L750MB Logger, you can customize “zones mapping” for all devices how will communicate with the SmartConnector. In CEF (Common Event Format) standard, the device zone is classified under “deviceZoneURI” and the SmartConnector zone is classified under “agentZoneURI“.

A zone represent a part of your network with contiguous IP addresses, for example LAN, DMZ, VPN, WIFI. If you customize your devices “zones mapping“, you will able to create, with your Logger, alerts, queries and reports for group of devices how are in the same zone. This will save you time 🙂

An ArcSight SmartConnector zone is represented by :

  • A starting IP address (for example : 192.168.0.15)
  • A ending IP address (for example : 192.168.0.20)
  • A zone name (for example : /All Zones/Office Zones/Printers)

The zone will be represented by this uncommented line :

192.168.0.15,192.168.0.20,/All Zones/Office Zones/Printers

In order to customize your devices “zones mapping“, you only have edit the “defaultzones.csv” file located in “$ARCSIGHT_HOME/current/user/agent/acp/” directory.

Delete the following line from the file :

#ignore.this.file <- delete this line

Then add your zones mapping, save the file and restart the SmartConnector.

ArcSight SmartConnector commands and features

If you have download for free the ArcSight Logger L750MB version, follow the installation guideline under Centos and install Windows Snare with ArcSight Syslog SmartConnector, you have now an operational lab or production environment. In this post we will describe you some SmartConnector commands and features. These commands and features are not documented in the provided ArcSight Logger L750MB documentation.

Starting the SmartConnector

If you don’t have specify, during the setup, to run the SmartConnector as a service, you will ne to start it manually. To start the SmartConnector you need to go in the “$ARCSIGHT_HOME/current/bin” directory and execute the “arcsight” script for Linux, or “arcsight.bat” script under Windows, with the following argument.

Starting ArcSight SmartConnector
Starting ArcSight SmartConnector

Once started, to confirm that the SmartConnector is working properly, you will have to check these outputs.

ArcSight SmartConnector starting outputs
ArcSight SmartConnector starting outputs

Eps” will give you the actual EPS throughput, and “Evts” the total number of events how have been processed by the SmartConnector. “ET” and “HT” should have twice the “Up” value in order to validate the the SmartConnector connexion with the Logger is working properly. If they are any communication troubles between the SmartConnector and the Logger you will have these kind of outputs.

ArcSight SmartConnector and Logger communication troubles
ArcSight SmartConnector and Logger communication troubles

Checking SmartConnector availability

To valide that the SmartConnector is up and running, you can use the following command.

ArcSight SmartConnector agent up
ArcSight SmartConnector agent up

If the SmartConnector is down, you will have this result.

ArcSight SmartConnector down
ArcSight SmartConnector down

This command will not validate that the communication between the SmartConnector and the Logger is up and running.

Restarting the SmartConnector

To restart the SmartConnector you will have to use the following command.

ArcSight SmartConnector restart
ArcSight SmartConnector restart

Stopping the SmartConnector

If you have start the SmartConnector in the standalone mode, a simple CTRL+C will terminate the activities. But you can also stop the activities with the following command :

Stopping ArcSight SmartConnector
Stopping ArcSight SmartConnector

Checking SmartConnector status

To check the complete SmartConnector status use the following command.

ArcSight SmartConnector status
ArcSight SmartConnector status

The output will provide you some useful informations about the SmartConnector activities (memory usage, agent type, agent version, processed events, EPS, last event processed date, etc.)

Checking SmartConnector DNS resolution

To verify that the SmartConnector is able to do DNS resolution you can execute the following command.

ArcSight SmartConnector DNS test
ArcSight SmartConnector DNS test

ArcSight Agent FlexAgent Regex Tester

ArcSight provide, with the SmartConnector, a tool how will permit you to create and test regex for your logs. SmartConnectors delivered for ArcSight Logger L750MB will not allow you to add FlexConnectors (custom agents), but you can still use this regex GUI for personal purposes. To start the regex GUI execute the following command.

ArcSight Agent FlexAgent regex tester
ArcSight Agent FlexAgent regex tester

For example, I have test the regex tool, with the following postfix log entry.

May 12 04:14:13 logger sendmail[3457]: p4C2EDU2003456: to=<[email protected]>, ctladdr=<[email protected]> (0/0), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=31483, dsn=2.0.0, stat=Sent

The regex tester will provide you a solution on how to parse this log.

ArcSight regex tester example
ArcSight regex tester example

ArcSight Logger L750MB – Syslog SmartConnector and Snare installation

In my previous blog post, I have detail how to install ArcSight Logger L750MB (the 49$ one) on a Centos 5.x. In this post I will explain you how to install a Windows Syslog ArcSight Connector, how to collect Windows events with Snare and how to push all these events to your Logger.

ArcSight Logger L750MB – network flows

As described in my “ArcSight Logger L750MB features and limits” blog post, this product version of ArcSight Logger has some limitations.

10 devices maximum could send they logs to the L750MB Logger. A device is the IP of the event generator. For example, you could have a Linux RHEL 5 server with Snort, the 2 products will be considered coming from the same source IP. Also, a device on the Logger, is a combination of a event source IP and a “Receiver“. We recommend you to use the same “Receiver” for common products on the same source IP.

With the L750MB version you will be allowed to install SmartConnectors to support these products :

  • Cisco PIX/ASA
  • Cisco IOS Routers and Switches
  • Juniper Network and Security Manager (NSM)
  • Juniper JUNOS Routers and Switches
  • Red Hat Enterprise Linux
  • SNARE
  • Snort

They are 2 provided SmartConnectors in ArcSight Download Center, one to install a Syslog server under Windows, and the other one to install a Syslog server under Linux. All the supported products should send they’re logs to the SmartConnector in Syslog 514/UDP or 514/TCP. When the products logs are received by the ArcSight SmartConnector, these logs are normalized in CEF (Common Event Format) and send encrypted to the Logger in SmartMessage format on port 9000/TCP. ArcSight SmartConnector is not only normalizing logs in CEF, but we will not explain in this blog post all the SmartConnectors capabilities. Here under a small representation of the network flows.

ArcSight Logger L750MB Network Flows
ArcSight Logger L750MB Network Flows

As you know, UDP is not a protocol what we can trust for delivering informations. UDP does not provide guarantee of delivery which can cause data to go missing. When considering connection problems or missing data, the TCP connection is much more desirable. Also, if you need to encrypt the data connection, you should use TCP. So we strongly recommend you to communicate with the ArcSight Syslog SmartConnector in TCP protocol. But the free version of Snare for Windows only support UDP protocol, so we will do this demonstration with UDP. If you want to have TCP support for Snare, you need to buy Snare server.

ArcSight Logger L750MB – Receiver configuration

First of all we need to create a “Receiver” on the Logger. This “Receiver” will be a SmartMessage one, in order to have the possibility to receive events in CEF.

To do this configuration, you only need to identify you on your Logger and to go in the “Configuration” menu, then click on the “Event Input/Output” sub-menu. The receiver will listen on port 9000/TCP, same port as for the Logger Web interface. We will name the receiver “SmartMessageReceiver01“, his type will be “SmartMessage Receiver” and his encoding “UTF8“.

ArcSight Logger L750MB SmartMessage Receiver Configuration part 1
ArcSight Logger L750MB SmartMessage Receiver Configuration

Once saved, you will find your configured receiver in the “Receivers” list. On the right of the receiver, in the receiver list, you need to active it by clicking on extreme right icon. When the receiver will be actif the extreme right icon will be replaced by the following one.

ArcSight Logger L750MB - Receiver startupWe will not pre-configure the devices, cause they will be auto-discovered by the Logger. And we will also not pre-configure a “Devices Group” and a “Storage Rule“.

ArcSight Syslog SmartConnector installation

Now we will install the Syslog SmartConnector on a dedicated box in order to centralize the communication from all the devices and to the Logger. The Syslog SmartConnector will use the TCP protocol. In our example, we will install the Windows Syslog SmartConnector (ArcSight-5.0.2.5703.0-Connector-Downloadable-Logger-Win.exe).

Execute the installation executable, and choose a proper installation folder, we recommend you to not install the SmartConnector on the C drive. Then choose a typical install and the installation executable will install the software and launch the SmartConnector wizard. Choose “ArcSight Logger SmartMessage (encrypted)” as destination and provide all the required informations for the SmartConnector and Logger interconnexion. The “Receiver Name” should be the same as declared on the Logger, here “SmartMessageReceiver01“.

ArcSight Logger L750MB SmartConnector interconnexion configuration
SmartConnector interconnexion configuration

The next screen will indicate you that a Syslog SmartConnector will be installed, and you will have to configure the syslog server by providing the listener IP address (here 192.168.178.66) and the related protocol (here UDP).

Syslog SmartConnector configuration
Syslog SmartConnector configuration

Now give a name (here WinSyslogSC01), a SmartConnector location (here Rack 10-01), a device location (here Rack 11) and a comment (here All rack 11 servers). These informations are optionals.

SmartConnector optional informations
SmartConnector optional informations

You will see in the last screen, that it is possible to reconfigure the SmartConnector with the “bin/runagentsetup.bat” script. Choose now if you want to install the SmartConnector as a standalone application or as a service, and if the service should start automatically.

Syslog SmartConnector as a service
Syslog SmartConnector as a service

The Syslog SmartConnector installation is now finished, but don’t forget to start the Syslog SmartConnector service 🙂

Snare Event Log Agent for Windows installation

Download Snare Event Log Agent for Windows and install it one every Windows server or station you want, but don’t forget that you are limited to 10 devices maximum.

Execute the installation binary, accept the agreement, choose an installation folder, let Snare to take over control of the EventLog configuration, enable the Snare remote control interface with a password and finish Snare installation. Launch “Snare for Windows” from you “Program” menu, you will be connected in the web remote control interface on port 6161/TCP.

Snare web remote control interface
Snare web remote control interface

In the “Network Configuration” menu, configure the “Destination Snare Server” (here 192.168.178.66), the “Destination Port” (here 514),  and click the checkbox for “Enable SYSLOG Header“, the save the configuration.

Snare for Windows configuration
Snare for Windows configuration

To reload the Snare configuration just click on the “Reload Settings” in the “Apply the Latest Audit Configuration“. And here we go, the Windows events are send to the Logger.  For further instructions on how to configure Snare we recommend you to read the Snare documentation 🙂

Windows Events in your Logger

In ArcSight Logger you will now have all the Windows Events directly accessible by the “Analyze” menu and “Search” sub-menu.

ArcSight Logger Windows Events Snare
ArcSight Logger Windows Events Snare