Tag Archives: ArcSight

ArcSight Logger File Receiver Configuration

ArcSight Logger propose different kind of receivers :

  • UDP receiver for UDP messages, such as SYSLOG.
  • TCP receiver for TCP messages, such as SYSLOG how can also be sent with TCP.
  • CEF UDP receiver for CEF (Common Event Format) messages sent through UDP.
  • CEF TCP receiver for CEF (Common Event Format) messages sent through TCP.
  • SmartMessage receiver for encrypted SmartMessage messages sent by SmartConnectors.
  • File Transfer to read remote logs using scp, sftp or ftp.
  • File Receiver to read logs from a local or remote file system such as NFS, CIFS or SAN.

In my previous blog posts, we configured SmartConnectors to send they’re messages to the Logger SmartMessage receiver. The SmartMessage reception method is the most used in an typical ArcSight Log Management infrastructure. With SmartConnectors and SmartMessage receiver usage you can benefit of :

  • SmartConnector normalization, categorization, aggregation, batching and filtering.
  • SmartMessage encrypted transmission of messages.

But it could happen that you need to collect events without SmartConnector or FlexConnector. For examples, maybe you have an existing network file system (NFS) to centralize all your Apache HTTP Server logs files, or maybe you are not allowed to install SmartConnector on a system and the policy doesn’t allow to send messages through UDP. File Receiver Logger receivers will provide you alternatives in order to still collect the events.

File Receiver Configuration

ArcSight Logger File Receiver allow you to read files from a network file system (NFS) CIFS, or storage area network (SAN). But the free ArcSight Logger L750MB only allow you File Receiver through a local share. So the ArcSight Logger L750MB processes, how are running under “arcsight” user and group should have the permissions to read and/or write into this share.

In order to setup a File Receiver for ArcSight Logger L750MB, first create a “filereceiver01” folder in “$ARCSIGHT_HOME“. Or you can mount a NFS, CIFS or SAN on the Logger filesystem.

Then log into the Logger Web interface and go into “Configuration -> Event Inpout/Output“. Click on the “Add” button, give a name to your File Receiver (for example: FileReceiver01), select “File Receiver” in the pull-down list and click on the “Next” button.

To complete the setup specify the following information’s :

  • RFS Names : On a L750MB Logger you can only select “LOCAL“, but on a appliance Logger you can select the previously declared NFS, CIFS or SAN share.
  • Folder : Specify the folder name, in our example “/home/arcsight/filereceiver01“.
  • Source Type : Select from the pull-down list your log file types. “Microsoft DHCP Log“, “Juniper Steel-Belted Radius“, “Apache HTTP Server Error“, “Apache HTTP Server Access“, “IBM DB2 Audit” or “Other“. The “Other” choice will allow you to import all kind of logs.
  • Wildcard (regex) : Allow you through regular expressions to describe the log file to read. “.*” mean all files.
  • Mode : Select from the pull-down list. With “Persist” mode, the Logger will remember which files have been processed and only processes them once. With “Rename” mode will rename the log file once it has been processed. With “Delete” mode, the file is deleted once it has been processed.
  • Rename extension : If you have select the “Rename” mode, specify the suffix to append to the log files. By default “.done“.
  • Character encoding : Select the log file character encoding.
  • Delay after seen : Specify the number of seconds to wait after a source file is first seen until it is processed. This allows the entire file to be copied before processing begins. By default “10” seconds.
  • Date/time locale : Select from the pull-down list your locale.
  • Date/time zone : Select from the pull-down list your time zone, if no time zone is specified into the log file. By default, the time zone is the Logger time zone.
  • Date/time loc. regex : Allow you through regular expressions to describe which characters represent the timestamps in the log file. By default no timestamp in the log file.
  • Date/time format : If timestamps are present in the log file, specify the format of the timestamps through format specifiers. By default no timestamps in the log file.
  • Event start (regex) : If you’re log files are multi-line, you can specify a regular expression how describe the start of a new event in the log file. By default, each line in the log file is considered as a single event.

Then save your configuration. You will have to active the File Receiver by clicking on the right button.

Directly after activating the File Receiver a Device will be created into “Configuration -> Devices“. In order to search easily on the File Receiver events, I recommend you to create a dedicated Device Group (ex : FileReceiver), from “Configuration -> Devices -> Device Groups“, and to associated the newly created File Receiver with this new Device Group.

Also I recommend you, to dedicate a Storage Group (ex : File Receivers) to all events coming from the File Receivers, from “Configuration -> Storage -> Storage Groups“. You can rename on existing Storage Group and adapt the associated retention period and maximum size.

Associate the created Device Group with the renamed Storage Group by Storage Rules, from “Configuration -> Storage -> Storage Rules).

Finally you have to restart the File Receiver by clicking twice on the right button. If you don’t restart the receiver, the Device and Storage Groups associated with the receiver are not active.

Retrieving events from File Receiver

If you have create dedicated Device Groups and/or Storage Groups for both File Receiver, you will be able to easily find the associated events through the Logger search engine.

For searches on Device Groups, just type the following command, where “FileReceiver” is you Device Group.

For searches on Storage Groups, just type the following command, where “File Receivers” is your Storage Group.

You can also mix all these search parameters.

Receivers Debugging

All activities related to the Logger Receivers are located in the “/home/arcsight/current/arcsight/logger/logs/logger_receiver.log” log file.

receiver_https” is corresponding to the SmartMessage Receiver, and “rfs_file_receiver” to the newly added File Receiver.

Eps(SLC)” is corresponding to the current EPS rate, “Total Events” is the total number of events the receiver has processed since the last restart, “Eps(max)” the maximum EPS rate the receiver has processed since the last restart, “Bps(SLC)” the bytes per second the receiver has processed.

ArcSight Logger L750MB now for free !

Good news for the Log Management market, ArcSight is now offering his downloadable software Logger L750MB version for free ! You don’t have to pay 49$ per year to install and fully use this Log Management solution. You can download this version from ArcSight website, but downloadable version can only be downloaded from the following countries: Australia, Belgium, Canada, Denmark, Finland, France, Germany, Hong Kong, India, Israel, Italy, Kuwait, Malaysia, Netherlands, Norway, Poland, Russia, Saudi Arabia, Singapore, South Africa, Spain, Sweden, Taiwan, Thailand, Turkey, United Arab Emirates, United Kingdom and United States.

If you want to install and play with ArcSight Logger L750MB, I have write some blogposts to help you :

ArcSight SmartConnectors silent mass upgrade

Since the Jun 2, ArcSight has release a new version of the free ArcSight Logger L750MB (5.1.0.5887.0) and for related SmartConnectors (5.1.3.5875.0). You can download these updates from ArcSight Download Center.

In my previous blogpost we have document on how silently mass install SmartConnectors, in this new blogpost we will see on how upgrade these SmartConnectors also silently. This blogpost is only applicable if you have standalone SmartConnectors, not connected to a Connectors Appliance, L3x00 Logger Appliance serie (whow integrate a Connectors Appliance) or directly connected to ArcSight ESM SIEM.

The previous installed SmartConnectors version was 5.0.2.5703.0, you can check your SmartConnector version by executing this command.

First of all you need to completely install, for example, a Windows Syslog SmartConnector as described in my previous “Syslog SmartConnector and Snare installation“.

After the SmartConnector 5.0.2.5703.0 installation start to install the 5.1.0.5887.0 version. You will see this following screen, just click “OK” to continue.

Later during the installation process, if you see the following screen just click on the “Cancel” button.

Open a command prompt and go to the SmartConnector installation directory (ex : C:\Program Files\ArcSightSmartConnectors) and execute the following command.

The “recorderui” start option will allow you to record the installation process in order to create a mass installation properties configuration file.

The installation wizard will propose you to select a “Silent Properties File Name” and a typical “Installation Target Folder“.

Create an “installer.properties” file on your desk and select it in the wizard. Also select a default installation folder, for example “D:\ArcSightSmartConnectors“. Now you can continue your typical SmartConnector configuration.

When you see this screen, check the “I do not want to change any settings” as described in the following screenshot.

Finish the setup, and you can examine your “installer.properties” file to adapt the properties with your needs.

Adapt the “USER_INSTALL_DIR” and “ARCSIGHT_AGENTSETUP_PROPERTIES” variables to your needs.

Now you can upgrade all your SmartConnectors in silent mode. If a properties file named either “installer.properties” reside in the same directory as the installer, it will automatically be used, overriding all other command line options, unless the “-f” option is used to point to another valid properties file.

The “-f” option can be used by following command line.

ArcSight SmartConnectors silent mass installation

With your free ArcSight L750MB Logger you can mass install ArcSight SmartConnectors with a silent properties configuration file. If you have to install, for example, 10 or more Syslog SmartConnectors, you will win time by reading this blog post.

First of all you need to create a properties configuration file template by installing a typical SmartConnector, with typical settings. Just start to install, for example, a Windows Syslog SmartConnector as described in my previous “Syslog SmartConnector and Snare installation“.

During the installation process, if you see the following screen just click on the “Cancel” button.

Open a command prompt and go to the SmartConnector installation directory (ex : C:\Program Files\ArcSightSmartConnectors) and execute the following command.

The “recorderui” start option will allow you to record the installation process in order to create a mass installation properties configuration file.

The installation wizard will propose you to select a “Silent Properties File Name” and a typical “Installation Target Folder“.

Create an “installer.properties” file on your desk and select it in the wizard. Also select a default installation folder, for example “D:\ArcSightSmartConnectors“. Now you can continue your typical SmartConnector configuration.

You can examine your “installer.properties” to adapt the properties with your needs.

For each SmartConnector you have to install you need to adapt into your “installer.properties” file :

– The SmartConnector name : AgentDetailsPanel.agentname
– The optional SmartConnector location : AgentDetailsPanel.agentlocation
– The optional device location : AgentDetailsPanel.devicelocation
– The optional comment : AgentDetailsPanel.comment

Now you can install all your SmartConnectors in silent mode. If a properties file named either “installer.properties” reside in the same directory as the installer, it will automatically be used, overriding all other command line options, unless the “-f” option is used to point to another valid properties file.

The “-f” option can be used by following command line.