Scalable realtime stats with Graphite

Introduction

---

Here at Adku, we’re always looking for ways to move faster and smarter.  There’s nothing worse than having to wait a day or two to see if a code push has a positive or negative effect on our bottom line.  One way to track application issues is to create a stat and graph it.  While there’s a litany of solutions out there today, we’re using Graphite, a realtime graphing framework and we love what its done for us thus far.  Here’s what Graphite can do:

http://graphite.wikidot.com/screen-shots

http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

We recently went through the process of setting up our own Graphite cluster and below are the references/steps that we used to get everything running.  We hope this helps :)

Graphite installation guide

---

This guide will install all the software packages required to get you up and running.  Graphite will be setup to run across multiple machines.  NOTE, these steps are virtually verbatim copies of our setup scripts.  We are assuming you’re working with a clean Ubuntu 10.04 LTS installation (we’re using Ubuntu AMIs on EC2)  We will NOT be delving into specifics for all non-Graphite related apps.  In case you’re curious, this guide will install the following packages:

Process control - supervisor - akin to init.d

Web serving - nginx, uwsgi - akin to apache + mod_wsgi/mod_python

Caching - memcached

Stats collection - statsite - akin to Etsy’s statsd without a dependency on node.js

Graphite - graphite-web, carbon, whisper - required for stats collection

Step 1 - Download our example configs

Step 2 - Run bootstrap.sh

Step 3 - Copy all files from the archive into /etc and /opt respectively

Step 4 - Update following config variables

/etc/nginx/nginx.conf - worker processes (line 4)

/etc/nginx/sites-enabled/graphite - public hostname (line 3)

/etc/supervisor/supervisord.conf - web processes (line 39)

/opt/graphite/conf/carbon.conf - storage dir and cluster servers (line 2 and line 96)

/opt/graphite/conf/relay-rules.conf - cluster servers (line 3)

/opt/graphite/conf/storage-schemas.conf - retention times (line 4 - optional)

/opt/graphite/webapp/graphite/local_settings.py - storage dir and cluster servers (all lines)

Step 5 - Setup the graphite-web - Django app

cd /opt/graphite/webapp/graphite

yes no | python manage.py syncdb

Step 6 - Spin up supervisord - should spin up all other processes

sudo supervisorctl status

sudo supervisord (only if needed)

Step 7 - Ensure that our processes didn’t blow up

sudo tail -f /var/log/supervisor/*.log

Step 8 - Feed stats to your cluster.  For a list of clients see

PHP / Java - https://github.com/etsy/statsd

Python - http://pypi.python.org/pypi/python-statsd

Ruby - https://github.com/dawanda/statsd-client

Step 9 - Check your stats server that your stat showed up!

http://statsd.yourhost.com:7001/

Further reference

---

Graphite

http://graphite.wikidot.com/screen-shots

http://readthedocs.org/docs/graphite/en/1.0/

nginx/uwsgi - used for web serving

http://wiki.nginx.org/Configuration

http://blog.martinfjordvald.com/2010/07/nginx-primer/

statsite - Python implementation of Etsy’s statsd

https://github.com/kiip/statsite

http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/