Wednesday, June 15, 2011

Working Remotely

At Adku, every engineer we hire gets a quad-core iMac with a 12GB memory upgrade and a 15-inch quad-core MacBook Pro with maximum memory upgrades (8GB).  When we're working remotely on our laptops, sometimes we'll realize that we forgot to check-in a piece of code that is sitting on our iMac at work.  If we could ssh into the iMac, we'd be able to check it in and continue where we left off.  The problem is our office is behind multiple NATs which make it hard to set up port forwarding and as a fast moving startup, it's not worth the time to set up our own name server.  We needed something quick and easy.  What we ended up doing was setting up SSH tunnels from our machines at work to an Amazon EC2 instance we call gateway.adku.com.  With the tunnel set up, accessing our machines at work is as simple as SSH'ing to the right port on gateway.adku.com.  Here is how to set one up for yourself.

    On Work Machine:
    # create security group and authorized ports
    # this assumes that you have an amazon ec2 account and 
    # ami-tools and api-tools setup.  
    # setting all that up is straightforward, but outside the scope of this post
    ec2-add-group --region us-east-1 gateway -d gateway
    ec2-authorize --region us-east-1 -p 22 gateway

    # create your gateway machine.
    # your ami may differ.  ubuntu likes to update its amis very often.  
    # also your keypair may have a different name, adjust accordingly
    ec2-run-instances ami-ee857587 --instance-type m1.small --region us-east-1 -k gsg-keypair-east -g gateway

    # note the public url of the newly created machine
    # also optionally setup an easier DNS name such as gateway.adku.com

    On Gateway Machine:
    sudo echo "GatewayPorts yes" >> /etc/ssh/sshd_config
    sudo /etc/init.d/ssh reload

    On Work Machine:
    # 20000 is an arbitrary port, you can choose any open port.  
    # you just have to remember it for later.
    # gateway.adku.com should also be replaced with your public dns name
    ec2-authorize --region us-east-1 -p 20000 gateway
    ssh -nNT -R gateway.adku.com:20000:localhost:22 gateway.adku.com

    And you're done!  

    Now you can get to your work machine from anywhere like this:

    On Any Machine:
    # 20000 is the port from earlier, make sure it matches

    Friday, May 27, 2011

    Giving back - Extending MRJob

    If you’ve ever used MRJob to run a MapReduce and wanted to debug a problem, you’ll soon realize that attaching a debugger to your mappers and reducers is very difficult. We’ve solved this by building a local, inline job runner that makes it as easy to debug your MapReduce as if it were a standalone Python script.

    Over the last few weeks, Adku has been playing around with a cool new project named MRJob, a python package that streamlines the development of MapReduce jobs. One of the core features of MRJob is that you can write your MapReduce job once (in Python!) and run it pretty much anywhere. Out of the box, MRJob comes with support for launching your jobs onto a Hadoop cluster. It also comes with built-in support for launching jobs directly on Amazon's ElasticMapReduce (EMR) service. Why is this exciting you ask? Well, Amazon makes it simple to launch arbitrarily large/small Hadoop clusters in a matter of minutes. If you need a tiny cluster to test that your job works remotely, you can run your cluster on a single “m1.small” instance. If you need a huge Hadoop cluster to crunch gigabytes or even terabytes worth of data, you can spin up a cluster with dozens of “m1.large” instances.

    Before spinning up hundreds of machines to run your mind-blowing MapReduce job, you'll probably want to test that your job works before blowing through your IT budget. Nothing sucks more than spinning up a huge cluster, waiting for Amazon to provision all your machines, then find out that you had a typo in your job. Because Amazon rounds up on instance-hours used, your 5 minute jaunt to nowhere just cost you “$/instance-hour * # of instances * 1 hour” Luckily, MRJob provides ways for you to test your job before launching on EMR.

    Out of the box, MRJob comes with support for a local job runner that acts like a mini-Hadoop cluster with a single mapper and a single reducer. The local runner works by first trying to emulate the environment you would see in Hadoop. Then it tries to run the equivalent of “python mapper -> sorter -> python reducer” Because the local job runner tries to clone the remote environment, this is for testing that you’ve uploaded all job dependencies. The local job runner tries to emulate Hadoop even more by using pipes to Hadoop streaming. Unfortunately, because the local job runner works by spinning up subprocesses, your favorite debugger will not be able to attach to your map/reduce phases.

    Over the last few weeks, we've struggled with debugging our local MapReduce jobs. Raising exceptions in the map/reduce phases feels gross. Emitting counters with entire error messages felt even worse. If you’ve ever tried to debug a mapreduce job in a debugger or an IDE, you’ll know how difficult it is to get something as simple as a traceback. Really, what we want do is attach a debugger to our map/reduce phase to see exactly what was going on. After some poking and prodding at MRJob's internals, we figured out a way to run mappers and reducers in the same process as the launcher script. By running your job with “--runner inline” you can now attach a debugger to your job and observe pretty much everything! “--runner inline” is designed to be used in conjunction with “--runner local” It is not meant to replace the default run mode (local) as we do NOT faithfully replicate the file system you would see in Hadoop. This runner is purely for development/debugging purposes.

    Where can you get this patch you ask? Well, we love giving back to the community so we sent our patch upstream a few days ago. Today, I’m happy to report that the maintainers over at Yelp accepted our changeset and has released as a part of mrjob 0.2.6!

    PyPi registration - http://pypi.python.org/pypi/mrjob

    Homepage - https://github.com/Yelp/mrjob
    Documentation - http://packages.python.org/mrjob/

    -Matt

    Monday, May 16, 2011

    We’ve moved into new offices!


    It's been really busy over the last few months, while we've been building our product and company, and we haven’t had much time to come up for air. We've got one piece of fun news that we'd like to share - we just moved into our new office! As we've expanded the team, we could no longer fit into our previous space which was graciously provided by our friends at Pivotal Labs.

    We’re based in the heart of the tech center of San Francisco in SoMa at 300 Beale St close to great technology startups and software giants as well as great restaurants and public transportation.

    We looked at quite a few different spaces and really wanted something unique and creative and, after a lot of looking, we found the perfect place for us. For those who’ve been around San Francisco for a while, this space used to be occupied Elroy’s which incidentally happened to be one of the first restaurants I ever ate at when I moved to the Bay Area.

    If you’re in the area, please stop by and say hello! We’ve got a lot of plans for the office and I’ve posted some pictures of how the space looks in the raw. If you have ideas on how we should decorate, just let us know in the comments.

    This is just another step in our journey as a startup and we’re really excited to share it with you.
    Entrance

    Bottom floor looking up at the mezzanine
    Front conference/game room

    Back conference room
    Future break room and bike racks
    Top floor looking down.

    Tuesday, February 8, 2011

    Startup Interviews

    I spent almost 6 years at Google interviewing nearly 300 potential Googlers. That inteviewing career started with an interviewing class and some shadowing of fellow engineers before diving in solo. At Google I focused my interview on trying to figure out if the person I was interviewing would have a good shot at being successful at Google. My requirements for Google success changed a bit over time but in general I looked for:
    1. Raw problem solving ability. I generally checked this by asking basically a math/problem solving question disguised as a coding question. It usually required almost no knowledge of any coding beyond what someone might learn in the first few months of coding.
    2. Ability to quickly see multiple solutions, weigh them, verbalize them, and select the most appropriate given the conditions that I've put forth. I'm constantly surprised at how rarely applicants verbalize the different options for solutions to a problem, outline the pros and cons of the solutions and then proceed.
    3. A substantial level of curiosity for anything they have spent time on. "If architectural decisions were made before you joined the team or without you, did you figure out why they were made, how would you have done things differently?", "How much time do you spend looking at library source files to really understand the technology you are using?", etc.
    4. Communication skills. This one is a bit trite but must be mentioned as it is as important as the others listed here. During an interview, the candidate's ability to articulate their thinking process, their background, and their goals is key.
    5. Ability to contribute to overall team happiness. If you are going to spend over 60% of your waking life during the week working, hopefully you enjoy what you are doing and who you are working with. When someone adds to that enjoyment, the result is a happy team, and happy teams are generally more productive, more communicative, and in the end, more successful.
    At Adku am I looking for anything different? I'm looking for every one of those qualities and more. The more:
    1. A self starter/entrepreneur personality. "Do you have side projects that you've worked on outside of school/work?", "Have you thought about starting your own company, why didn't you?" I believe the best hires for a startup are entrepreneurs or almost-entrepreneurs.
    2. A coding fearlessness of the unknown. A startup generally won't have an entire team devoted to just one slice of the entire stack. Because of this, it is likely that anyone hired will have to dive into any part of the stack and be extremely productive. If the person can't pick up a new language and/or technology and get a Hello World working within hours, then they are likely the wrong person.
    3. A level of self awareness and honesty. Being a part of a startup is a wild and amazing ride. Efficiency is built not only on the ability to code and code review quickly, but on the ability to recognize issues and course correct as fast as possible. Being successful is based on many things, including your ability to be honest about your mistakes. Accessing this in an interview is difficult and subtle, but it can be done. Applicants, shouldn't "gloss" over details that you are unfamiliar with in the hopes that the interviewer won't realize that you don't have a deep understanding; admit what you understand and what you don't.
    4. Product strategy/tech business curiosity. Product discussion is a very good thing and having every member of the team engaged is that discussion is important. The ability to have a healthy discourse on product/company strategy, both from business and technical point of views, is key.
    Does this sound like you? If so, we'd love to talk to you!

    Wednesday, February 2, 2011

    HBase vs Cassandra

    Adku recently decided to migrate part of it's data to a NoSQL database in order to deal with increasing load on our MySQL database. We evaluated many options including MongoDB, Amazon SimpleDB, and a few others, but ultimately narrowed the options to just Cassandra and HBase. We experimented with both databases and evaluated them more deeply and ultimately decided to use Cassandra. This post explains the high level reasons why we chose Cassandra.

    Disclaimer: While these decisions apply to Adku, they might not apply to your situation. Always do your own investigation and experimentation before choosing any large part of your system.
    1. Reliability
      - We wrote a stress test tool to simulate how the databases would behave under high load. We used the default minimalistic configuration for each database following their respective documentations. Our stress test inserted 1 million rows of the full "Alice in Wonderland" text (~180kb) inserted into each row done continuously with one concurrent thread into a 3 node cluster. Surprisingly, the HBase region servers actually crashed on us consistently. Cassandra never crashed once. Although we would obviously scale the cluster under high loads and crashing nodes, this is still definitely concerning and a large win for Cassandra.

    2. Performance
      - We also wrote a simulation tool to test a more realistic scenario to see how each database would respond. Similar to the above, our tool inserted 1 million rows with 2000 characters of ascii text inserted into each row done continuously with one concurrent thread into a 3 node cluster. HBase averaged 507 microseconds per write where Cassandra averaged 480 microseconds per write. We interpreted this as basically equivalent performance so there was no real winner here.

    3. Consistency
      - Consistency is not a hard requirement for our specific use case so Cassandra's eventual consistency model is fine for us. If we do end up needing consistency, Cassandra can support it using their configurable CAP model, we would just have to take a performance hit. HBase has consistency so it is technically the winner here, but since it's not a hard requirement for us, it doesn't carry much weight.

    4. Single Point of Failure
      - Hadoop's namenode which HBase depends on is a single point of failure. This means that if the namenode goes down, the entire database is unreachable. All Cassandra nodes are identical so there is no single point of failure. This is a win for Cassandra.

    5. Hot Spot Problem
      - Our relevant row keys are currently all timestamps. HBase chooses the node to store data on by row key in sorted order. Cassandra by default stores them on a random node in random order. This means that HBase will fall into a hotspot problem where one node is handling most of the write traffic. Cassandra, however, distributes the load across all nodes evenly. This is a win for Cassandra.

    6. MapReduce
      - HBase is built on top of HDFS and Hadoop. This means that MapReduce is very easy. Cassandra supports MapReduce, but doesn't support streaming MapReduce so you have to write them in Java. I'm also not sure what the relative performance is. Cassandra supports data locality so that MapReduces tasks end up processing data on the same machine as the MapReduce task so it's possible that performance is comparable, but I haven't done adequate tests. HBase is the winner here, but so far we haven't seen any drawbacks to running MapReduce on Cassandra (besides having to write in Java).

    7. Simpler, Hackable
      - Cassandra is a simpler implementation and much easier to hack. This is the same reason why we chose Tornado instead of Apache as our web server and we've actually made quite a few modifications to the Tornado web server as a result. Bugs are also much easier to debug. HBase by comparison is much more complicated and harder to debug and hack. This is a win for Cassandra and we've even already submitted patches back into open source Cassandra.

    8. Community Support
      - As of today, there are 175 users in the #cassandra channel on irc.freenode.net. HBase by comparison only has 74. Aside from IRC, it does appear that the Cassandra community is larger and more helpful than HBase. Another win for Cassandra.

    Tuesday, January 25, 2011

    Code Reviews

    As an engineer coming from Google where every line of code is reviewed by another engineer before it is submitted, I had assumed that there would be a publicly available open source tool to help us facilitate code reviews in a similar fashion outside of the Googleplex. As it turns out, there are very few available and almost all are for doing code reviews post-commit. This includes the code review tool built into Google Code Hosting. Google Code recommends that you submit to a branch and then request a code review of the branch before merging into trunk. While that method certainly works, it's cumbersome and the overhead is too high. What inevitably ends up happening is nobody ends up requesting code reviews and the whole process gets dropped.

    In my experience, the only way to make sure code reviews actually happen is to enforce it in your tools. But that's a post for another day. We don't enforce code reviews here at Adku because we don't feel that code reviews should be mandatory just yet for a three person startup, but you can bet that once we get bigger, we will.

    Anyway, after hours of searching and experimenting with various tools, we settled on Rietveld which was developed by Guido van Rossum, author of Python and Mondrian. The problem with Rietveld, however, is that it doesn't work out of the box! It is meant to be run on appengine, but only supports Django 1.1 so you have to change your appengine setup to use Django 1.1 and this is not a trivial process. Moreover, if you run Rietveld on appengine, most of your changes will be available publicly with no way to restrict access to your internal team without diving in and modifying Rietveld source code.

    In the end, after a bit of troubleshooting, we successfully got Rietveld up and running and even set it up internally to restrict access.

    Now that it's up, it is beautiful, works great, and is exactly what we were looking for. Typical use looks something like this
    1. make some local edits
      $ vi my_source_code.py
    2. send the changes to carlos for code review
      $ python rietveld/upload.py --send_mail -r carlos
    3. carlos gets an email that looks like this
      From:
      Date: Thu, Nov 4, 2010 at 11:19 AM
      Subject: better demo (issue24)
      To: carlos@adku.com


      Reviewers: carlos,

      Please review this at http://rietveld:8000/24/

      Affected files:
      ...
      ...
    4. carlos clicks on the link and sees the unsubmitted changes in a web ui with change highlighting, syntax highlighting, and an inline commenting ui
    5. carlos enters a few comments inline with the changes and hits send
    6. i make the changes suggested and re-send the code
    7. he reviews again and hits approve
    8. i submit the pending change in my local client

    Monday, January 3, 2011

    Peer reviews in a 4 month old, 3 person startup?

    Even though Adku is only 4 months old, part of our company culture involves doing periodic NxN peer reviews. Everyone reviews everyone else and provides hard, candid feedback on strengths and weaknesses. But are peer reviews necessary for a 3 person startup that's only 4 months old you ask? Yes, it's critical, and I'll tell you why.

    Company culture is very hard to change

    Company culture is one of those things you need to engrain as early as you can. What you do in the beginning months of your startup will permeate through it's entire life and it's very hard to change. That's why we decided to have regular retrospective meetings to improve how we work, that's why we work on puzzles together every week, and that's why we take trips together and drink together.

    Conflict resolution is critical to company success

    The most common reason for company dissolution in the early stages is some kind of personal conflict. If the early team can not discuss issues like product direction options or annoying work habits openly and resolve them, then they are doomed to fail. I've seen this time and again. Peer reviews help make sure issues get raised, discussed, and resolved.

    Deep insight into team strengths will help shape team responsibilities

    Typically, engineering teams inside companies have managers. This is because managers add value. Don't get me wrong, there are certainly managers out there who decrease value, but in general, they exist because they can assign responsibilities intelligently and make adjustments based on team dynamics and individual strengths. Without a manager, the founding team has to do this collaboratively. Peer reviews are essential to make this happen.

    Each individual's personal growth creates shareholder value

    The output you create at your company today may be worth $500,000/year. If peer reviews end up making you more productive resulting in an increased output worth $600,000/year, then you've just increased the value of the company. To be fair, chances are you will also get a raise, but it won't be equal to your increase in output. This is because you yourself do not actually produce $600,000/year. You are leveraged by the rest of the company. An engineer at Google may output $10,000,000/year, but may only output $100,000/year at Microsoft for equivalent work.