Log management comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc.). Log management covers log collection, centralized aggregation, long-term retention, log analysis (in real-time and in bulk after storage) as well as log search and reporting. Log management is driven by reasons of security, system and network operations (such as system or network administration) and regulatory compliance. Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs). Wikipedia definition.
Once you have install and configure your SYSLOG ArcSight SmartConnector to communicate with your free L750MB Logger, you can customize “zones mapping” for all devices how will communicate with the SmartConnector. In CEF (Common Event Format) standard, the device zone is classified under “deviceZoneURI” and the SmartConnector zone is classified under “agentZoneURI“.
A zone represent a part of your network with contiguous IP addresses, for example LAN, DMZ, VPN, WIFI. If you customize your devices “zones mapping“, you will able to create, with your Logger, alerts, queries and reports for group of devices how are in the same zone. This will save you time 🙂
An ArcSight SmartConnector zone is represented by :
A starting IP address (for example : 192.168.0.15)
A ending IP address (for example : 192.168.0.20)
A zone name (for example : /All Zones/Office Zones/Printers)
The zone will be represented by this uncommented line :
In order to customize your devices “zones mapping“, you only have edit the “defaultzones.csv” file located in “$ARCSIGHT_HOME/current/user/agent/acp/” directory.
If you don’t have specify, during the setup, to run the SmartConnector as a service, you will ne to start it manually. To start the SmartConnector you need to go in the “$ARCSIGHT_HOME/current/bin” directory and execute the “arcsight” script for Linux, or “arcsight.bat” script under Windows, with the following argument.
Starting ArcSight SmartConnector
Once started, to confirm that the SmartConnector is working properly, you will have to check these outputs.
ArcSight SmartConnector starting outputs
“Eps” will give you the actual EPS throughput, and “Evts” the total number of events how have been processed by the SmartConnector. “ET” and “HT” should have twice the “Up” value in order to validate the the SmartConnector connexion with the Logger is working properly. If they are any communication troubles between the SmartConnector and the Logger you will have these kind of outputs.
ArcSight SmartConnector and Logger communication troubles
Checking SmartConnector availability
To valide that the SmartConnector is up and running, you can use the following command.
ArcSight SmartConnector agent up
If the SmartConnector is down, you will have this result.
ArcSight SmartConnector down
This command will not validate that the communication between the SmartConnector and the Logger is up and running.
Restarting the SmartConnector
To restart the SmartConnector you will have to use the following command.
ArcSight SmartConnector restart
Stopping the SmartConnector
If you have start the SmartConnector in the standalone mode, a simple CTRL+C will terminate the activities. But you can also stop the activities with the following command :
Stopping ArcSight SmartConnector
Checking SmartConnector status
To check the complete SmartConnector status use the following command.
ArcSight SmartConnector status
The output will provide you some useful informations about the SmartConnector activities (memory usage, agent type, agent version, processed events, EPS, last event processed date, etc.)
Checking SmartConnector DNS resolution
To verify that the SmartConnector is able to do DNS resolution you can execute the following command.
ArcSight SmartConnector DNS test
ArcSight Agent FlexAgent Regex Tester
ArcSight provide, with the SmartConnector, a tool how will permit you to create and test regex for your logs. SmartConnectors delivered for ArcSight Logger L750MB will not allow you to add FlexConnectors (custom agents), but you can still use this regex GUI for personal purposes. To start the regex GUI execute the following command.
ArcSight Agent FlexAgent regex tester
For example, I have test the regex tool, with the following postfix log entry.
ArcSight propose you, until first June 2011, to download for free it’s ArcSight Logger L750MB version. If you register the free version of ArcSight Logger L750MB with the “FreeLogger4WebA” promo code, you will also receive a “Love they logs” t-shirt.
We have several blog posts how could help you install and configure your free ArcSight Logger.
In my previous blog post, I have detail how to install ArcSight Logger L750MB (the 49$ one) on a Centos 5.x. In this post I will explain you how to install a Windows Syslog ArcSight Connector, how to collect Windows events with Snare and how to push all these events to your Logger.
10 devices maximum could send they logs to the L750MB Logger. A device is the IP of the event generator. For example, you could have a Linux RHEL 5 server with Snort, the 2 products will be considered coming from the same source IP. Also, a device on the Logger, is a combination of a event source IP and a “Receiver“. We recommend you to use the same “Receiver” for common products on the same source IP.
With the L750MB version you will be allowed to install SmartConnectors to support these products :
Cisco PIX/ASA
Cisco IOS Routers and Switches
Juniper Network and Security Manager (NSM)
Juniper JUNOS Routers and Switches
Red Hat Enterprise Linux
SNARE
Snort
They are 2 provided SmartConnectors in ArcSight Download Center, one to install a Syslog server under Windows, and the other one to install a Syslog server under Linux. All the supported products should send they’re logs to the SmartConnector in Syslog 514/UDP or 514/TCP. When the products logs are received by the ArcSight SmartConnector, these logs are normalized in CEF (Common Event Format) and send encrypted to the Logger in SmartMessage format on port 9000/TCP. ArcSight SmartConnector is not only normalizing logs in CEF, but we will not explain in this blog post all the SmartConnectors capabilities. Here under a small representation of the network flows.
ArcSight Logger L750MB Network Flows
As you know, UDP is not a protocol what we can trust for delivering informations. UDP does not provide guarantee of delivery which can cause data to go missing. When considering connection problems or missing data, the TCP connection is much more desirable. Also, if you need to encrypt the data connection, you should use TCP. So we strongly recommend you to communicate with the ArcSight Syslog SmartConnector in TCP protocol. But the free version of Snare for Windows only support UDP protocol, so we will do this demonstration with UDP. If you want to have TCP support for Snare, you need to buy Snare server.
ArcSight Logger L750MB – Receiver configuration
First of all we need to create a “Receiver” on the Logger. This “Receiver” will be a SmartMessage one, in order to have the possibility to receive events in CEF.
To do this configuration, you only need to identify you on your Logger and to go in the “Configuration” menu, then click on the “Event Input/Output” sub-menu. The receiver will listen on port 9000/TCP, same port as for the Logger Web interface. We will name the receiver “SmartMessageReceiver01“, his type will be “SmartMessage Receiver” and his encoding “UTF8“.
Once saved, you will find your configured receiver in the “Receivers” list. On the right of the receiver, in the receiver list, you need to active it by clicking on extreme right icon. When the receiver will be actif the extreme right icon will be replaced by the following one.
We will not pre-configure the devices, cause they will be auto-discovered by the Logger. And we will also not pre-configure a “Devices Group” and a “Storage Rule“.
ArcSight Syslog SmartConnector installation
Now we will install the Syslog SmartConnector on a dedicated box in order to centralize the communication from all the devices and to the Logger. The Syslog SmartConnector will use the TCP protocol. In our example, we will install the Windows Syslog SmartConnector (ArcSight-5.0.2.5703.0-Connector-Downloadable-Logger-Win.exe).
Execute the installation executable, and choose a proper installation folder, we recommend you to not install the SmartConnector on the C drive. Then choose a typical install and the installation executable will install the software and launch the SmartConnector wizard. Choose “ArcSight Logger SmartMessage (encrypted)” as destination and provide all the required informations for the SmartConnector and Logger interconnexion. The “Receiver Name” should be the same as declared on the Logger, here “SmartMessageReceiver01“.
SmartConnector interconnexion configuration
The next screen will indicate you that a Syslog SmartConnector will be installed, and you will have to configure the syslog server by providing the listener IP address (here 192.168.178.66) and the related protocol (here UDP).
Syslog SmartConnector configuration
Now give a name (here WinSyslogSC01), a SmartConnector location (here Rack 10-01), a device location (here Rack 11) and a comment (here All rack 11 servers). These informations are optionals.
SmartConnector optional informations
You will see in the last screen, that it is possible to reconfigure the SmartConnector with the “bin/runagentsetup.bat” script. Choose now if you want to install the SmartConnector as a standalone application or as a service, and if the service should start automatically.
Syslog SmartConnector as a service
The Syslog SmartConnector installation is now finished, but don’t forget to start the Syslog SmartConnector service 🙂
Snare Event Log Agent for Windows installation
Download Snare Event Log Agent for Windows and install it one every Windows server or station you want, but don’t forget that you are limited to 10 devices maximum.
Execute the installation binary, accept the agreement, choose an installation folder, let Snare to take over control of the EventLog configuration, enable the Snare remote control interface with a password and finish Snare installation. Launch “Snare for Windows” from you “Program” menu, you will be connected in the web remote control interface on port 6161/TCP.
Snare web remote control interface
In the “Network Configuration” menu, configure the “Destination Snare Server” (here 192.168.178.66), the “Destination Port” (here 514), and click the checkbox for “Enable SYSLOG Header“, the save the configuration.
Snare for Windows configuration
To reload the Snare configuration just click on the “Reload Settings” in the “Apply the Latest Audit Configuration“. And here we go, the Windows events are send to the Logger. For further instructions on how to configure Snare we recommend you to read the Snare documentation 🙂
Windows Events in your Logger
In ArcSight Logger you will now have all the Windows Events directly accessible by the “Analyze” menu and “Search” sub-menu.
We have 
We will not pre-configure the devices, cause they will be auto-discovered by the Logger. And we will also not pre-configure a “Devices Group” and a “Storage Rule“.
