Adku recently decided to migrate part of it's data to a NoSQL database in order to deal with increasing load on our MySQL database. We evaluated many options including MongoDB, Amazon SimpleDB, and a few others, but ultimately narrowed the options to just Cassandra and HBase. We experimented with both databases and evaluated them more deeply and ultimately decided to use Cassandra. This post explains the high level reasons why we chose Cassandra.
Disclaimer: While these decisions apply to Adku, they might not apply to your situation. Always do your own investigation and experimentation before choosing any large part of your system.
- Reliability
- We wrote a stress test tool to simulate how the databases would behave under high load. We used the default minimalistic configuration for each database following their respective documentations. Our stress test inserted 1 million rows of the full "Alice in Wonderland" text (~180kb) inserted into each row done continuously with one concurrent thread into a 3 node cluster. Surprisingly, the HBase region servers actually crashed on us consistently. Cassandra never crashed once. Although we would obviously scale the cluster under high loads and crashing nodes, this is still definitely concerning and a large win for Cassandra. - Performance
- We also wrote a simulation tool to test a more realistic scenario to see how each database would respond. Similar to the above, our tool inserted 1 million rows with 2000 characters of ascii text inserted into each row done continuously with one concurrent thread into a 3 node cluster. HBase averaged 507 microseconds per write where Cassandra averaged 480 microseconds per write. We interpreted this as basically equivalent performance so there was no real winner here. - Consistency
- Consistency is not a hard requirement for our specific use case so Cassandra's eventual consistency model is fine for us. If we do end up needing consistency, Cassandra can support it using their configurable CAP model, we would just have to take a performance hit. HBase has consistency so it is technically the winner here, but since it's not a hard requirement for us, it doesn't carry much weight. - Single Point of Failure
- Hadoop's namenode which HBase depends on is a single point of failure. This means that if the namenode goes down, the entire database is unreachable. All Cassandra nodes are identical so there is no single point of failure. This is a win for Cassandra. - Hot Spot Problem
- Our relevant row keys are currently all timestamps. HBase chooses the node to store data on by row key in sorted order. Cassandra by default stores them on a random node in random order. This means that HBase will fall into a hotspot problem where one node is handling most of the write traffic. Cassandra, however, distributes the load across all nodes evenly. This is a win for Cassandra. - MapReduce
- HBase is built on top of HDFS and Hadoop. This means that MapReduce is very easy. Cassandra supports MapReduce, but doesn't support streaming MapReduce so you have to write them in Java. I'm also not sure what the relative performance is. Cassandra supports data locality so that MapReduces tasks end up processing data on the same machine as the MapReduce task so it's possible that performance is comparable, but I haven't done adequate tests. HBase is the winner here, but so far we haven't seen any drawbacks to running MapReduce on Cassandra (besides having to write in Java). - Simpler, Hackable
- Cassandra is a simpler implementation and much easier to hack. This is the same reason why we chose Tornado instead of Apache as our web server and we've actually made quite a few modifications to the Tornado web server as a result. Bugs are also much easier to debug. HBase by comparison is much more complicated and harder to debug and hack. This is a win for Cassandra and we've even already submitted patches back into open source Cassandra. - Community Support
- As of today, there are 175 users in the #cassandra channel on irc.freenode.net. HBase by comparison only has 74. Aside from IRC, it does appear that the Cassandra community is larger and more helpful than HBase. Another win for Cassandra.
Glad that you found a solution that works for you.
ReplyDeleteYour analysis of #4, "Single Point of Failure", contains some inaccuracies, though.
You state, regarding HBase, "This means that when the master goes down, although the system can automatically recover, this recovery takes on the order of seconds during which your database is down."
This simply is not true. The master is not involved at all in handling of normal client requests. All data in the cluster will continue to be available, for both reads and writes, in the event of a master failure. The cluster can continue operating without any master at all. The only operations impacted will be HBase internal operations, such as region splits.
On the Cassandra is simpler than HBase, I have to ask, by what measure?
ReplyDeleteIs it by this measure:
cassandra$ find test -iname \*.java | xargs wc -l | tail -1
7011 total
cassandra$ cd ../hbase_trunk
hbase_trunk ryan$ find src/test -iname \*.java | xargs wc -l | tail -1
52090 total
Because it seems like you might be throwing the baby out with the bathwater on #7 there.
By chance is this Ryan T of FB fame?
DeleteHi Gary H, You're absolutely right. I confused the HBase master with the Hadoop namenode. I've edited the post to correct the mistake. Thanks!
ReplyDeleteHi Ryan, We called Cassandra "simpler" because it doesn't depend on Hadoop, HDFS, and ZooKeeper which means fewer moving parts.
ReplyDeleteInteresting article - thanks. I am curious - what version of HBase did you use and how did you have the cluster configured?
ReplyDeleteHey Robert, We used HBase 0.20.6 and we followed the configuration instructions in these two articles: hbashttp://hbase.apache.org/notsoquick.html
ReplyDeletehttp://blog.ibd.com/howto/experience-installing-hbase-0-20-0-cluster-on-ubuntu-9-04-and-ec2/
Unfortunately, I don't think I saved the actual configuration files we ended up with, but they were extremely minimal.
Thanks Jesse.
ReplyDeleteAny measure of performance without lots of concurrency is not an accurate benchmark of HBase or Cassandra.
ReplyDeleteHey David,
ReplyDeleteI would agree, but it should be able to handle a single thread just as well. Setting up a test with lots of concurrency was part of our plan, but after HBase crashed on a single thread, we basically short-circuited the performance tests.
Besides, we weren't looking to do a very deep exhaustive benchmark of the technologies as other sites have already done, we just wanted to get a feel for which would work best for us in our situation.
Jesse
Hi~~
ReplyDeleteWhat version of Cassandra, Hbase do you test?
and How many cluster do you use?
thank you~~~
Hey CharSyam,
ReplyDeleteCassandra was version 0.7.
HBase was version 0.20.6.
Our tests used a single cluster of three machines each.
Jesse
Hi Jesse,
ReplyDeletejust to clarify the performance: 500ms per write, 1 million records: so it took approx 6 days to run the test? They seem very slow ?
Hi Mogol,
ReplyDeleteIt's in microseconds, not milliseconds.
Microseconds!
ReplyDeleteOn number 8, I am on both email lists cassandra and hbase and I get way more email on hbase(I am not on the irc channels though). ie. maybe it's hard to tell...I thought the hbase community was bigger until you mentioned the irc channels(maybe it's more of which community is using what for means of communication and both communities are the same size?)
ReplyDeleteoh, and another good reference is how many companies search for people who know hadoop vs. how many search for cassandra(if you add java, it is essentially a tie ;) )...
ReplyDeletehttp://www.indeed.com/jobtrends?q=hadoop%2C+cassandra&l=
DataStax Enterprise (DSE) includes a Cassandra-enabled Hive/Pig MapReduce client.
ReplyDeletehttp://www.datastax.com/docs/1.0/datastax_enterprise/about_hive
DataStax Enterprise (DSE) is a commercial distribution of Apache Cassandra and Apache Hadoop developed by DataStax. DSE provides Hadoop MapReduce capabilities using CassandraFS, an HDFS-compatible storage layer inside of Cassandra. By replacing HDFS with CassandraFS, users are able to leverage their current MapReduce jobs on Cassandra’s peer-to-peer, fault-tolerant, and scalable architecture. DataStax Enterprise is also able to support dual workloads, allowing you to use the same cluster of machines for both real-time applications and data analytics without having to move the data around between systems.
I wonder how much memory they used during stress tests and performance tests. Did you checked those or have any idea about approximate usages?
ReplyDeleteHi, I dont see any read performance metrics in your test. Is there any reason for skipping them?
ReplyDeleteHi,
ReplyDeletenice comparison, we did something similar, but excluded HBase and Cassandra, because they "looked" complex at a time.
at the end we evaluated Cassandra too, but did not evaluated HBase.
Why we choose Apache Cassandra over Riak and Tokyo Cabinet
http://nmmm.nu/cass.htm