3 ways to push data to graylog2

If you are a sysadmin or developer and you haven’t heard of graylog2 then your missing out.Graylog2 takes log data(or what ever you want to throw at it), stores it for you and allows you to search it.It does this by using mongodb as its backend and providing a web interface written in rails to categorize and search it.In my case its very useful. I manage servers in 4 physical locations, slice host, rackspace, rackspace cloud and EC2. I needed a way to keep all of the system logs in one place with out having to work too hard at it.Graylog2 was my solution.

So far I use 3 different methods to write data to graylog2.

  1. rsyslog over UDP
  2. piping data over net cat
  3. Using the GELF gem which is specific to graylog2

(1) rsyslog over UDPThis is the easiest one by far, and used to write system log data.On ubuntu all I had to do was disable syslog, enable rsyslog and add this one line to /etc/rsyslog.conf

*.*       @graylog2.posterfoo.com

Thats all I had to do.BTW if you want to send the same data over TCP do the following instead.

*.*       @@graylog2.posterfoo.com

(2) piping data over net catThis one is also easy to use, just pipe data to net cat provided with a logging facility and hostname.In the example below I am piping a log file to facility 7(debug) with from the hostname foo.foo.com

#!/bin/sh  tail -F -q /var/log/nginx/accesslog |   while read -r line ; do  echo "<7> foo.foo.com $line" | nc -w 1 -u graylog2.posterfoo.com 514  done

Thats it.Once in graylog2 you can sort/search by hostname, logging level or regex on the data itself.

(3) Using the GELF gem which is specific to graylog2This method provides the most flexibility in that you are allowed to create custom fields.In the example below I am parsing the access_log before I submit to graylog2 using the GELF gem.This results in custom fields which can be used to categorize and sort such as method(GET,PUT,etc..), uri, size, referrer, etc…

#!/usr/bin/ruby  require 'rubygems'  require 'gelf'  def send_gelf(ip,method,uri,code,size,referral)  line = ip + " " + method + " " + uri + " " + code + " " + size + " " + referral  n = GELF::Notifier.new("graylog2.posterfoo.com", 12201)  n.notify!(:host => "prod-nginx", :level => 1, :short_message => line, :_ip => ip, :_method => method, :_uri => uri, :_code =>   code, :_size => size, :_referral => referral)  end  ARGF.each do |line|  x = line.split(/s+/)  send_gelf(x[0],x[7],x[8],x[10],x[11],x[12])  end
About these ads

6 thoughts on “3 ways to push data to graylog2

  1. Thanks for your post, it’s very helpful to me.But I dont know to using GELF to forward log to graylog2. Can you explain how, many thanks

  2. All Installation and configuration steps with troubleshooting Tips for Graylog2http://sharmith.blogspot.in/2012/05/network-device-syslog-ng-logstash.htmlhttp://sharmith.blogspot.in/2012/06/Installation-of-Elasticsearch-and-MongoDB.htmlhttp://sharmith.blogspot.in/2012/07/ruby-regex-syntax_26.htmlhttp://sharmith.blogspot.in/2012/08/graylog2-complete-suite-on-fedora-14-17.html

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s