How to create custom graphs with Munin

Munin is lacking many features that cacti has but one thing its really good at is creating custom graphs.Basically all you need is a script written in any language that when run will print out the values and when given the config argument will print the config for the graph.In the example below I am graphing the number of unicorn processes running on a box and the number of that are busy.The values:

./unicorn_inuse cap.value 21inuse.value 11

You can see above I am getting 2 values to graph, cap.value is the total number of unicorn processes running and inuse.value is the number that are busy.

The config:

./unicorn_inuse configgraph_title Total Unicorns in useinuse.type GAUGEinuse.label Unicorns in useinuse.draw LINE1graph_category Unicorngraph_args --base 1000 -l 0graph_scale nocap.label Total Unicornscap.draw LINE2cap.type GAUGE

Not too many details in the config but graph_category is how to put graphs in a specific bucket in the munin UI.

The graph:The code:

#!/usr/bin/env ruby  def get_total()  cmd = 'ps aux| grep capuser | grep unicorn | wc -l'  output = `#{cmd}`  num = output.match(/d+/)  return numenddef get_chillin()  cmd = "ps aux| grep capuser | grep unicorn | grep 'chillin'| wc -l"  output = `#{cmd}`  num = output.match(/d+/)  return numenddef config()       puts 'graph_title Total Unicorns in use'      puts 'inuse.type GAUGE'     puts 'inuse.label Unicorns in use'  puts 'inuse.draw LINE1'  puts 'graph_category Unicorn'     puts 'graph_args --base 1000 -l 0'      puts 'graph_scale no'    puts 'cap.label Total Unicorns'  puts 'cap.draw LINE2'  puts 'cap.type GAUGE'end    argu =  ARGV[0]     if argu == 'config'       config()     else       total = get_total()      chillin = get_chillin()      inuse = total[0].to_i - chillin[0].to_i  puts "cap.value " + total[0].to_s       puts "inuse.value " + inuse.to_s     end

3 ways to push data to graylog2

If you are a sysadmin or developer and you haven’t heard of graylog2 then your missing out.Graylog2 takes log data(or what ever you want to throw at it), stores it for you and allows you to search it.It does this by using mongodb as its backend and providing a web interface written in rails to categorize and search it.In my case its very useful. I manage servers in 4 physical locations, slice host, rackspace, rackspace cloud and EC2. I needed a way to keep all of the system logs in one place with out having to work too hard at it.Graylog2 was my solution.

So far I use 3 different methods to write data to graylog2.

1. rsyslog over UDP
2. piping data over net cat
3. Using the GELF gem which is specific to graylog2

(1) rsyslog over UDPThis is the easiest one by far, and used to write system log data.On ubuntu all I had to do was disable syslog, enable rsyslog and add this one line to /etc/rsyslog.conf

*.*       @graylog2.posterfoo.com

Thats all I had to do.BTW if you want to send the same data over TCP do the following instead.

*.*       @@graylog2.posterfoo.com

(2) piping data over net catThis one is also easy to use, just pipe data to net cat provided with a logging facility and hostname.In the example below I am piping a log file to facility 7(debug) with from the hostname foo.foo.com

#!/bin/sh  tail -F -q /var/log/nginx/accesslog |   while read -r line ; do  echo "<7> foo.foo.com $line" | nc -w 1 -u graylog2.posterfoo.com 514  done

Thats it.Once in graylog2 you can sort/search by hostname, logging level or regex on the data itself.

(3) Using the GELF gem which is specific to graylog2This method provides the most flexibility in that you are allowed to create custom fields.In the example below I am parsing the access_log before I submit to graylog2 using the GELF gem.This results in custom fields which can be used to categorize and sort such as method(GET,PUT,etc..), uri, size, referrer, etc…

#!/usr/bin/ruby  require 'rubygems'  require 'gelf'  def send_gelf(ip,method,uri,code,size,referral)  line = ip + " " + method + " " + uri + " " + code + " " + size + " " + referral  n = GELF::Notifier.new("graylog2.posterfoo.com", 12201)  n.notify!(:host => "prod-nginx", :level => 1, :short_message => line, :_ip => ip, :_method => method, :_uri => uri, :_code =>   code, :_size => size, :_referral => referral)  end  ARGF.each do |line|  x = line.split(/s+/)  send_gelf(x[0],x[7],x[8],x[10],x[11],x[12])  end