Tuesday, October 11, 2011

Speed feels good

Sitting in front of my computer typing at over 120 words-per-minute feels DAMN good.. damn good. or at least I imagine it would be if I could actually type 120wpm. Sadly, as much as I try, I can't type that fast, but.. I *can* cheat my way into working as fast as my fictional friend, let's call him Carlos, can (or faster, shhh!).

There are probably better methods of working faster out there, but in this post, I want to convince you that you should alias your already short commands to be even shorter and how doing that will make both your fingers more efficient AND your brain more efficient.

Efficient Fingers

How many times a day do you type "ls"? 100? 1000? If you
you would save 33% of the number of keystrokes used to issue that command (including the enter key). 33% is pretty damn significant. That's 100-1000 keystrokes less per day. Your fingers are already looking like body builders you type-a-holic, they don't need to be working out any harder than they need to.

What about "cd .."?
increases your productivity by 300%! That's like 200 less keystrokes a day?

Not to belabor the point, but how many times do you type "cd" and don't immediately type "ls" afterwards?

Here are some common ones I use daily.

You may have noticed that I chose "d" and "s" for these common commands. The reason is because "d" and "s" are on the home row of your qwerty keyboard. This means your fingers have a shorter distance to travel which means MORE SPEED. They are also for your left hand so that your right hand can sit on top of the enter key ready and waiting so that it can get all up on that enter key RIGHT AFTER you type your 1-letter command. 2 hands are better than 1.

You can alias all day long and find longer and longer commands that you type often. Things like "ssh -i ec2-keypair ubuntu@ec2.adku.com" or "cd /mnt/var/log/supervisor" each of which I do 100 times a day have their own aliases, saving me completely UNFATHOMABLE amounts of time. I seriously can't even comprehend the savings.

Anyway, the point is, (fictional) Carlos may type faster than me, but that doesn't mean he gets more done than me =)

Efficient Brain

In practice, after years of obsessing over aliases and reducing keystrokes, I've unexpectedly found that the productivity gain is actually quite a bit more than just the reduced mechanical overhead of hitting keys. It also allows you to work that much closer to the speed of thought, eliminating the costly context switch of deciding what to do vs executing the commands. We all know how expensive context switching is when writing code, but there is a much smaller, but more pervasive context switch that happens between the time you decide you want to look at your log files and when your actually able to see them. Sometimes you get to the right folder and forget what you were looking for. Getting to your log files in 3 seconds instead of 10 seconds turns out to be a big difference and lets you keep your mind on what your doing and less on telling your fingers what buttons to push == SPEED x 1000!

If you like working fast and getting a lot done, consider applying for a job at Adku.

Monday, October 3, 2011

Scalable realtime stats with Graphite


Here at Adku, we’re always looking for ways to move faster and smarter.  There’s nothing worse than having to wait a day or two to see if a code push has a positive or negative effect on our bottom line.  One way to track application issues is to create a stat and graph it.  While there’s a litany of solutions out there today, we’re using Graphite, a realtime graphing framework and we love what its done for us thus far.  Here’s what Graphite can do:

We recently went through the process of setting up our own Graphite cluster and below are the references/steps that we used to get everything running.  We hope this helps :)

Graphite installation guide

This guide will install all the software packages required to get you up and running.  Graphite will be setup to run across multiple machines.  NOTE, these steps are virtually verbatim copies of our setup scripts.  We are assuming you’re working with a clean Ubuntu 10.04 LTS installation (we’re using Ubuntu AMIs on EC2)  We will NOT be delving into specifics for all non-Graphite related apps.  In case you’re curious, this guide will install the following packages:

Process control - supervisor - akin to init.d
Web serving - nginx, uwsgi - akin to apache + mod_wsgi/mod_python
Caching - memcached
Stats collection - statsite - akin to Etsy’s statsd without a dependency on node.js
Graphite - graphite-web, carbon, whisper - required for stats collection

Step 2 - Run bootstrap.sh

Step 3 - Copy all files from the archive into /etc and /opt respectively

Step 4 - Update following config variables

/etc/nginx/nginx.conf - worker processes (line 4)
/etc/nginx/sites-enabled/graphite - public hostname (line 3)
/etc/supervisor/supervisord.conf - web processes (line 39)
/opt/graphite/conf/carbon.conf - storage dir and cluster servers (line 2 and line 96)
/opt/graphite/conf/relay-rules.conf - cluster servers (line 3)
/opt/graphite/conf/storage-schemas.conf - retention times (line 4 - optional)
/opt/graphite/webapp/graphite/local_settings.py - storage dir and cluster servers (all lines)

Step 5 - Setup the graphite-web - Django app

cd /opt/graphite/webapp/graphite
yes no | python manage.py syncdb

Step 6 - Spin up supervisord - should spin up all other processes

sudo supervisorctl status
sudo supervisord (only if needed)

Step 7 - Ensure that our processes didn’t blow up

sudo tail -f /var/log/supervisor/*.log

Step 8 - Feed stats to your cluster.  For a list of clients see

Step 9 - Check your stats server that your stat showed up!


Further reference


nginx/uwsgi - used for web serving

statsite - Python implementation of Etsy’s statsd