In this post we will do a presentation of Graylog2 features, components and installation details on Centos 5.
Graylog2 features and components presentation
Graylog2 is a free and open source log management solution that provide you a centralized repository and access to all your infrastructures logs. All the logs are stored in MongoDB, a scalable and high-performance database.
Graylog2 is composed of a server written in Java that will accept Syslog messages via TCP, UDP or AMQP (Advanced Message Queuing Protocol). AMPQ is an open standard for Messaging Middleware that allow different platforms in different languages to send messages to one another. Graylog2 is also using Drools Expert to evaluate all incoming messages against user defined rule file.
Graylog2 Web interface will allow you to search through the logs, filter them, blacklist out certain messages and create “streams“. An unlimited number of user can access the Web interface how will be able only to read defined and subscribed “streams“. Graylog2 Web interface also provide a way to use Nagios to check of the number of new log messages exceeds a given maximum.
Graylog2 “streams” are saved searches that permit you to quickly access to an overview of specifics occurrences. You can forward your “streams” to other endpoints through UDP Syslog, GELF or to Loggly, a cloud log management solution. GELF (Graylog Extended Log Format) will allow you to by pass Syslog limitations (message length, structure, timeouts, connection troubles) for your applications and servers. “Streams” will also allow you to send alarms when the number of new message reaching a given maximum during a given period. All users who subscribed to the “stream alarms” or to the “stream” will get an email alarm.
Graylog2 Centos 5 installation
In order to have a complete and functional Graylog2 log management solution we have to install three main components, MongoDB, graylog2-server and graylog2-web-interface.
MongoDB database installation
MongoDB propose to Centos and Fedora users yum-installable RPM packages for x86 and x86_64 platforms. “mongo-10gen” (mongodb client) and “mongo-10gen-server” (mongodb server) are available through the 10gen MongoDB repository. Just follow the “Centos and Fedora Packages” documentation to allow you server to install these packages. Then simply execute the following command to install MongoDB server and client.
$ sudo yum install mongo-10gen-server
MongoDB configuration file is located in “/etc/mongod.conf” and the associated sysconfig file is in “/etc/sysconfig/mongod“. When started MongoDB will run under mongod user and group.
First edit the MongoDB configuration file and change “nohttpinterface = false” to “nohttpinterface = false“. Then start MongoDB server with the following command.
$ sudo /etc/init.d/mongod start
Now we will create a user and password to allow Graylog2 to connect to MongoDB server. The database will directly be created during the user creation, and the database will be stored in “/var/lib/mongo“. To connect you to MongoDB server you have only to run the “mongo” client.
$ sudo mongo
> use graylog2
> db.addUser(“login”, “password”)
We have now a user (login) created, with his associated password, for database graylog2. If we wish to perform further operations we need to execute the following command.
> db.auth(“login”, “password”)
We can view existing users for the database with the following command.
> db.system.users.find()
For further security and authentication configurations please follow the MongoDB documentation.
We need now to configure the MongoDB server default listener port (27017/TCP). Just uncomment “port = 27017” line in the MongoDB configuration file. MongoDB will listen on the loopback (127.0.0.1). Also we need to turn on security for authentication by uncommenting the “auth = true” line.
Now restart MongoDB with the following command.
$ sudo /etc/init.d/mongod restart
Graylog2 server installation
Graylog2 server require to install openjdk.
$ sudo yum install openjdk
Download Graylog2 server from Github and adapt the following commands to your need.
$ sudo cp graylog2-server-0.9.5p1.tar.gz /opt/
$ sudo cd /opt$ sudo tar -zxvf graylog2-server-0.9.5p1.tar.gz
$ sudo ln -s graylog2-server-0.9.5p1 graylog2
$ sudo cd graylog2
We need to have the graylog2 server configuration file in “/etc/” folder.
$ sudo cp graylog2.conf.example /etc/graylog2.conf
In “/etc/graylog2.conf” configuration file change all “mongodb*” settings with your MongoDB configuration. For example :
# MongoDB Configuration
mongodb_useauth = true
mongodb_user = login
mongodb_password = password
mongodb_host = localhost
#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019
mongodb_database = graylog2
mongodb_port = 27017
Also configure, in the same configuration file, the Syslog server listener port and protocol. By default the Syslog server is listening on 514/UDP.
Now start Graylog2 server with the following command.
$ sudo cd /opt/graylog2/bin/
$ sudo ./graylog2ctl start
To stop graylog2 server execute the following command.
$ sudo ./graylog2ctl stop
Graylog2 Web interface installation
Graylog2 Web interface is running under Ruby, so we first need to install the latest version of Ruby. Please remove all your previous Ruby installation cause Centos 5 only support an old release of Ruby how is not compatible with Graylog2 and other dependencies.
$ sudo yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs
Make sure you have all the required development tools :
$ sudo yum install openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel
Download the latest Ruby sources and proceed with installing :
$ ./configure –enable-shared –enable-pthread –prefix=/usr
$ make
$ sudo make install
Ruby 1.9.2 and above now includes RubyGems so there’s no need to install it separately.
Test that everything installed successfully :
Update all the gems, install git and rake
$ sudo gem update && gem install git rake
Now download Graylog2 Web interface from Github and adapt to your needs the following commands.
$ sudo cp graylog2-web-interface-0.9.5p2.tar.gz /opt/
$ sudo cd /opt
$ sudo tar -zxvf graylog2-web-interface-0.9.5p2.tar.gz
$ sudo ln -s graylog2-web-interface-0.9.5p2 graylog2-web-interface
$ sudo cd graylog2-web-interface
We have also to install bundler with the following commands.
$ sudo gem install bundler
$ sudo bundle install
Edit all “*.yml” configuration files in “/opt/graylog2-web-interface/config/” folder.
“email.yml” configuration file will contain all required email configurations for alarms.
“general.yml” configuration file will contain all Graylog2 server general configurations such as hostname, automatic Graylog2 version check, etc.
“mongoid.yml” configuration file will contain all MongoDB configurations. For example :
production:
host: localhost
port: 27017
username: login
password: password
database: graylog2
We will server Graylog2 Web interface through Apache and Passenger.
To install Passanger just run the following command :
$ sudo gem install passenger
$ sudo passenger-install-apache2-module
$ sudo chown -R apache:apache /opt/graylog2-web-interface-0.9.5p2
$ sudo chown -R apache:apache /opt/graylog2-web-interface
Create a “passenger.conf” file in “/etc/httpd/conf.d/” directory and add the following entries :
LoadModule passenger_module /usr/lib/ruby/gems/1.9.1/gems/passenger-3.0.7/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.9.1/gems/passenger-3.0.7
PassengerRuby /usr/bin/ruby
The in your “httpd.conf” file include the “passenger.conf” file.
Add a Virtual Host in your “httpd.conf“, for example :
ServerName xxx.xxxx.com
DocumentRoot /opt/graylog2-web-interface/public
Allow from all
Options -MultiViews
ErrorLog /var/log/httpd/xxx.xxx.com_error.log
LogLevel warn
CustomLog /var/log/httpd/xxx.xxx.com_access.log combined
Also include the “passenger.conf” in the “httpd.conf” file and restart apache :
“Include conf.d/passenger.conf”
Now you will be able to connect you on your vhost, configure the Graylog2 first user and connect into the Web interface.
