3 ways to push data to graylog2

If you are a sysadmin or developer and you haven’t heard of graylog2 then your missing out.Graylog2 takes log data(or what ever you want to throw at it), stores it for you and allows you to search it.It does this by using mongodb as its backend and providing a web interface written in rails to categorize and search it.In my case its very useful. I manage servers in 4 physical locations, slice host, rackspace, rackspace cloud and EC2. I needed a way to keep all of the system logs in one place with out having to work too hard at it.Graylog2 was my solution.

So far I use 3 different methods to write data to graylog2.

1. rsyslog over UDP
2. piping data over net cat
3. Using the GELF gem which is specific to graylog2

(1) rsyslog over UDPThis is the easiest one by far, and used to write system log data.On ubuntu all I had to do was disable syslog, enable rsyslog and add this one line to /etc/rsyslog.conf

*.*       @graylog2.posterfoo.com

Thats all I had to do.BTW if you want to send the same data over TCP do the following instead.

*.*       @@graylog2.posterfoo.com

(2) piping data over net catThis one is also easy to use, just pipe data to net cat provided with a logging facility and hostname.In the example below I am piping a log file to facility 7(debug) with from the hostname foo.foo.com

#!/bin/sh  tail -F -q /var/log/nginx/accesslog |   while read -r line ; do  echo "<7> foo.foo.com $line" | nc -w 1 -u graylog2.posterfoo.com 514  done

Thats it.Once in graylog2 you can sort/search by hostname, logging level or regex on the data itself.

(3) Using the GELF gem which is specific to graylog2This method provides the most flexibility in that you are allowed to create custom fields.In the example below I am parsing the access_log before I submit to graylog2 using the GELF gem.This results in custom fields which can be used to categorize and sort such as method(GET,PUT,etc..), uri, size, referrer, etc…

#!/usr/bin/ruby  require 'rubygems'  require 'gelf'  def send_gelf(ip,method,uri,code,size,referral)  line = ip + " " + method + " " + uri + " " + code + " " + size + " " + referral  n = GELF::Notifier.new("graylog2.posterfoo.com", 12201)  n.notify!(:host => "prod-nginx", :level => 1, :short_message => line, :_ip => ip, :_method => method, :_uri => uri, :_code =>   code, :_size => size, :_referral => referral)  end  ARGF.each do |line|  x = line.split(/s+/)  send_gelf(x[0],x[7],x[8],x[10],x[11],x[12])  end

Advertisements