In this post we will do a presentation of Graylog2 features, components and installation details on Centos 5.
Graylog2 features and components presentation
Graylog2 is a free and open source log management solution that provide you a centralized repository and access to all your infrastructures logs. All the logs are stored in MongoDB, a scalable and high-performance database.
Graylog2 is composed of a server written in Java that will accept Syslog messages via TCP, UDP or AMQP (Advanced Message Queuing Protocol). AMPQ is an open standard for Messaging Middleware that allow different platforms in different languages to send messages to one another. Graylog2 is also using Drools Expert to evaluate all incoming messages against user defined rule file.
Graylog2 Web interface will allow you to search through the logs, filter them, blacklist out certain messages and create “streams“. An unlimited number of user can access the Web interface how will be able only to read defined and subscribed “streams“. Graylog2 Web interface also provide a way to use Nagios to check of the number of new log messages exceeds a given maximum.
Graylog2 “streams” are saved searches that permit you to quickly access to an overview of specifics occurrences. You can forward your “streams” to other endpoints through UDP Syslog, GELF or to Loggly, a cloud log management solution. GELF (Graylog Extended Log Format) will allow you to by pass Syslog limitations (message length, structure, timeouts, connection troubles) for your applications and servers. “Streams” will also allow you to send alarms when the number of new message reaching a given maximum during a given period. All users who subscribed to the “stream alarms” or to the “stream” will get an email alarm.
Graylog2 Centos 5 installation
In order to have a complete and functional Graylog2 log management solution we have to install three main components, MongoDB, graylog2-server and graylog2-web-interface.
MongoDB database installation
MongoDB propose to Centos and Fedora users yum-installable RPM packages for x86 and x86_64 platforms. “mongo-10gen” (mongodb client) and “mongo-10gen-server” (mongodb server) are available through the 10gen MongoDB repository. Just follow the “Centos and Fedora Packages” documentation to allow you server to install these packages. Then simply execute the following command to install MongoDB server and client.
$ sudo yum install mongo-10gen-server
MongoDB configuration file is located in “/etc/mongod.conf” and the associated sysconfig file is in “/etc/sysconfig/mongod“. When started MongoDB will run under mongod user and group.
First edit the MongoDB configuration file and change “nohttpinterface = false” to “nohttpinterface = false“. Then start MongoDB server with the following command.
$ sudo /etc/init.d/mongod start
Now we will create a user and password to allow Graylog2 to connect to MongoDB server. The database will directly be created during the user creation, and the database will be stored in “/var/lib/mongo“. To connect you to MongoDB server you have only to run the “mongo” client.
$ sudo mongo
> use graylog2
> db.addUser(“login”, “password”)
We have now a user (login) created, with his associated password, for database graylog2. If we wish to perform further operations we need to execute the following command.
> db.auth(“login”, “password”)
We can view existing users for the database with the following command.
> db.system.users.find()
For further security and authentication configurations please follow the MongoDB documentation.
We need now to configure the MongoDB server default listener port (27017/TCP). Just uncomment “port = 27017” line in the MongoDB configuration file. MongoDB will listen on the loopback (127.0.0.1). Also we need to turn on security for authentication by uncommenting the “auth = true” line.
Now restart MongoDB with the following command.
$ sudo /etc/init.d/mongod restart
Graylog2 server installation
Graylog2 server require to install openjdk.
$ sudo yum install openjdk
Download Graylog2 server from Github and adapt the following commands to your need.
$ sudo cp graylog2-server-0.9.5p1.tar.gz /opt/
$ sudo cd /opt$ sudo tar -zxvf graylog2-server-0.9.5p1.tar.gz
$ sudo ln -s graylog2-server-0.9.5p1 graylog2
$ sudo cd graylog2
We need to have the graylog2 server configuration file in “/etc/” folder.
$ sudo cp graylog2.conf.example /etc/graylog2.conf
In “/etc/graylog2.conf” configuration file change all “mongodb*” settings with your MongoDB configuration. For example :
# MongoDB Configuration
mongodb_useauth = true
mongodb_user = login
mongodb_password = password
mongodb_host = localhost
#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019
mongodb_database = graylog2
mongodb_port = 27017
Also configure, in the same configuration file, the Syslog server listener port and protocol. By default the Syslog server is listening on 514/UDP.
Now start Graylog2 server with the following command.
$ sudo cd /opt/graylog2/bin/
$ sudo ./graylog2ctl start
To stop graylog2 server execute the following command.
$ sudo ./graylog2ctl stop
Graylog2 Web interface installation
Graylog2 Web interface is running under Ruby, so we first need to install the latest version of Ruby. Please remove all your previous Ruby installation cause Centos 5 only support an old release of Ruby how is not compatible with Graylog2 and other dependencies.
$ sudo yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs
Make sure you have all the required development tools :
$ sudo yum install openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel
Download the latest Ruby sources and proceed with installing :
$ ./configure –enable-shared –enable-pthread –prefix=/usr
$ make
$ sudo make install
Ruby 1.9.2 and above now includes RubyGems so there’s no need to install it separately.
Test that everything installed successfully :
Update all the gems, install git and rake
$ sudo gem update && gem install git rake
Now download Graylog2 Web interface from Github and adapt to your needs the following commands.
$ sudo cp graylog2-web-interface-0.9.5p2.tar.gz /opt/
$ sudo cd /opt
$ sudo tar -zxvf graylog2-web-interface-0.9.5p2.tar.gz
$ sudo ln -s graylog2-web-interface-0.9.5p2 graylog2-web-interface
$ sudo cd graylog2-web-interface
We have also to install bundler with the following commands.
$ sudo gem install bundler
$ sudo bundle install
Edit all “*.yml” configuration files in “/opt/graylog2-web-interface/config/” folder.
“email.yml” configuration file will contain all required email configurations for alarms.
“general.yml” configuration file will contain all Graylog2 server general configurations such as hostname, automatic Graylog2 version check, etc.
“mongoid.yml” configuration file will contain all MongoDB configurations. For example :
production:
host: localhost
port: 27017
username: login
password: password
database: graylog2
We will server Graylog2 Web interface through Apache and Passenger.
To install Passanger just run the following command :
$ sudo gem install passenger
$ sudo passenger-install-apache2-module
$ sudo chown -R apache:apache /opt/graylog2-web-interface-0.9.5p2
$ sudo chown -R apache:apache /opt/graylog2-web-interface
Create a “passenger.conf” file in “/etc/httpd/conf.d/” directory and add the following entries :
LoadModule passenger_module /usr/lib/ruby/gems/1.9.1/gems/passenger-3.0.7/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.9.1/gems/passenger-3.0.7
PassengerRuby /usr/bin/ruby
The in your “httpd.conf” file include the “passenger.conf” file.
Add a Virtual Host in your “httpd.conf“, for example :
ServerName xxx.xxxx.com
DocumentRoot /opt/graylog2-web-interface/public
Allow from all
Options -MultiViews
ErrorLog /var/log/httpd/xxx.xxx.com_error.log
LogLevel warn
CustomLog /var/log/httpd/xxx.xxx.com_access.log combined
Also include the “passenger.conf” in the “httpd.conf” file and restart apache :
“Include conf.d/passenger.conf”
Now you will be able to connect you on your vhost, configure the Graylog2 first user and connect into the Web interface.
How do you configure a replica_set????
There must be an error on the following line: (change x to y with exact same value ?)
First edit the MongoDB configuration file and change “nohttpinterface = false” to “nohttpinterface = false“. Then start MongoDB server with the following command.
“nohttpinterface = false” to “nohttpinterface = false“
Do you mean taking out the “#”. When I did, mongod gives an error,
Starting mongod: warning: remove or comment out this line by starting it with ‘#’, skipping now : nohttpinterface=false
Good howto, got most of it working but have an issue.
When going to http://localhost:9200 I get prompted to open a file of type application/json, I tried pointing it at /usr/bin/java but it didn’t work, what have I done wrong?
Very much a newb at this!
change “nohttpinterface = false” to “nohttpinterface = false“
Is this a typo? Do you know what it should read?
Howto worked flawlessly, thanks!
“Make sure your mongoid.yml is using valid yml syntax. You need two spaces before host, port … ”
From: https://groups.google.com/group/graylog2/browse_thread/thread/2f1c0e80c97ce517
It took me about 4 hours to figure this out – to say that making spacing important in a config is stupid would be an understatement….
HTH someone.
Hi,
I use this howto to install graylog2 on a CentOS 6 server, everything is ok except at the end when I try to launch the web interface, I can see ‘Internal Server Error’ in my browser and this error in the graylog error log file:
*** Exception Mongoid::Errors::InvalidDatabase in application (Database should be a Mongo::DB, not NilClass.)
So I created an initializer mongo.rb:
/opt/graylog2-web-interface-0.9.5p2/config/initializers/mongo.rb
mongoid_conf = YAML::load_file(Rails.root.join('config/mongoid.yml'))[Rails.env]
Mongoid.configure do |config|
config.master = Mongo::Connection.new(mongoid_conf['host'],
mongoid_conf['port']).db(mongoid_conf['database'])
end
Then in the browser I see a page with error message: Ruby(rack) application could not be started and this error in the graylog error log file:
*** Exception NoMethodError in PhusionPassenger::Rack::ApplicationSpawner (undefined method `[]' for nil:NilClass)
I use CentOS 6 with ruby 1.9.3p0, mongoid 2.0.1, passenger 3.0.9, rack 1.2.2 and rails 3.0.6
Anyone can help me ? Thanks
Hi, and thanks for the reply
You for it with the first one – it was the SELinux permissions stuffing Passenger – turned it off (didn’t even know it was installed byd efault and enabled!), rebooted the box, and viola – all started working!
Thanks for your help – now I can see if I can get the logging working!
Cheers!
DaZZa
Hello Dazza,
Do you have check the rights on /usr/lib/ruby/gems/1.9.1/gems/passenger-3.0.8/agents/PassengerWatchdog and subfolders ?
And also do you have check your SELinux configuration.
http://groups.google.com/group/phusion-passenger/browse_thread/thread/a9a55422ad8771e1
Also other ressources :
http://www.google.com/search?hl=en&source=hp&q=Unable+to+start+the+Phusion+Passenger&btnG=Recherche+Google&meta=&aq=f&aqi=&aql=&oq=&gs_rfai=#pq=unable+to+start+the+phusion+passenger&hl=en&sugexp=gsnos%2Cn%3D2&cp=57&gs_id=8&xhr=t&q=unable+to+start+the+phusion+passenger+watchdog+permission+denied+13&pf=p&sclient=psy&source=hp&pbx=1&oq=Unable+to+start+the+Phusion+Passenger+Permission+denied+(&aq=0b&aqi=g-b1&aql=f&gs_sm=&gs_upl=&bav=on.2,or.r_gc.r_pw.&fp=2c09157da83efc36&biw=1421&bih=695
Regards
Regards
I’m having the same issue – I get errors in the server error log which read as follows
[error] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog (/usr/lib/ruby/gems/1.9.1/gems/passenger-3.0.8/agents/PassengerWatchdog): Permission denied (13)
I’ve followed the instructions above exactly, as far as I can tell. I’m also getting the following error in the server logs relating to accessing the site
[error] [client 10.50.12.198] Directory index forbidden by Options directive: /opt/graylog2-web-interface/public/
which basically dumps me to the default “Yeah, the Apache Server is installed” web page.
Any ideas welcomed, because I really want to get Graylog working!
Hi,
I am follwed what you described and everything works fine graylog2-server and mongodb but graylog2-web-interface. When i access it through the url it just show the public folder index. I can browse the folder but it not runs the server. What am i misssing please help.
Cheers
Peter
Hi Peter,
What are the error in your apache error logs ?
Regards