Wednesday, February 2, 2011

HBase vs Cassandra

Adku recently decided to migrate part of it's data to a NoSQL database in order to deal with increasing load on our MySQL database. We evaluated many options including MongoDB, Amazon SimpleDB, and a few others, but ultimately narrowed the options to just Cassandra and HBase. We experimented with both databases and evaluated them more deeply and ultimately decided to use Cassandra. This post explains the high level reasons why we chose Cassandra.

Disclaimer: While these decisions apply to Adku, they might not apply to your situation. Always do your own investigation and experimentation before choosing any large part of your system.
  1. Reliability
    - We wrote a stress test tool to simulate how the databases would behave under high load. We used the default minimalistic configuration for each database following their respective documentations. Our stress test inserted 1 million rows of the full "Alice in Wonderland" text (~180kb) inserted into each row done continuously with one concurrent thread into a 3 node cluster. Surprisingly, the HBase region servers actually crashed on us consistently. Cassandra never crashed once. Although we would obviously scale the cluster under high loads and crashing nodes, this is still definitely concerning and a large win for Cassandra.

  2. Performance
    - We also wrote a simulation tool to test a more realistic scenario to see how each database would respond. Similar to the above, our tool inserted 1 million rows with 2000 characters of ascii text inserted into each row done continuously with one concurrent thread into a 3 node cluster. HBase averaged 507 microseconds per write where Cassandra averaged 480 microseconds per write. We interpreted this as basically equivalent performance so there was no real winner here.

  3. Consistency
    - Consistency is not a hard requirement for our specific use case so Cassandra's eventual consistency model is fine for us. If we do end up needing consistency, Cassandra can support it using their configurable CAP model, we would just have to take a performance hit. HBase has consistency so it is technically the winner here, but since it's not a hard requirement for us, it doesn't carry much weight.

  4. Single Point of Failure
    - Hadoop's namenode which HBase depends on is a single point of failure. This means that if the namenode goes down, the entire database is unreachable. All Cassandra nodes are identical so there is no single point of failure. This is a win for Cassandra.

  5. Hot Spot Problem
    - Our relevant row keys are currently all timestamps. HBase chooses the node to store data on by row key in sorted order. Cassandra by default stores them on a random node in random order. This means that HBase will fall into a hotspot problem where one node is handling most of the write traffic. Cassandra, however, distributes the load across all nodes evenly. This is a win for Cassandra.

  6. MapReduce
    - HBase is built on top of HDFS and Hadoop. This means that MapReduce is very easy. Cassandra supports MapReduce, but doesn't support streaming MapReduce so you have to write them in Java. I'm also not sure what the relative performance is. Cassandra supports data locality so that MapReduces tasks end up processing data on the same machine as the MapReduce task so it's possible that performance is comparable, but I haven't done adequate tests. HBase is the winner here, but so far we haven't seen any drawbacks to running MapReduce on Cassandra (besides having to write in Java).

  7. Simpler, Hackable
    - Cassandra is a simpler implementation and much easier to hack. This is the same reason why we chose Tornado instead of Apache as our web server and we've actually made quite a few modifications to the Tornado web server as a result. Bugs are also much easier to debug. HBase by comparison is much more complicated and harder to debug and hack. This is a win for Cassandra and we've even already submitted patches back into open source Cassandra.

  8. Community Support
    - As of today, there are 175 users in the #cassandra channel on irc.freenode.net. HBase by comparison only has 74. Aside from IRC, it does appear that the Cassandra community is larger and more helpful than HBase. Another win for Cassandra.

Posted by Jesse Shieh at 9:16 AM

51 comments:

  1. Glad that you found a solution that works for you.

    Your analysis of #4, "Single Point of Failure", contains some inaccuracies, though.

    You state, regarding HBase, "This means that when the master goes down, although the system can automatically recover, this recovery takes on the order of seconds during which your database is down."

    This simply is not true. The master is not involved at all in handling of normal client requests. All data in the cluster will continue to be available, for both reads and writes, in the event of a master failure. The cluster can continue operating without any master at all. The only operations impacted will be HBase internal operations, such as region splits.

    ReplyDelete

  2. On the Cassandra is simpler than HBase, I have to ask, by what measure?

    Is it by this measure:
    cassandra$ find test -iname \*.java | xargs wc -l | tail -1
    7011 total
    cassandra$ cd ../hbase_trunk
    hbase_trunk ryan$ find src/test -iname \*.java | xargs wc -l | tail -1
    52090 total

    Because it seems like you might be throwing the baby out with the bathwater on #7 there.

    ReplyDelete

  3. Hi Gary H, You're absolutely right. I confused the HBase master with the Hadoop namenode. I've edited the post to correct the mistake. Thanks!

    ReplyDelete

  4. Hi Ryan, We called Cassandra "simpler" because it doesn't depend on Hadoop, HDFS, and ZooKeeper which means fewer moving parts.

    ReplyDelete

  5. Interesting article - thanks. I am curious - what version of HBase did you use and how did you have the cluster configured?

    ReplyDelete

  6. Hey Robert, We used HBase 0.20.6 and we followed the configuration instructions in these two articles: hbashttp://hbase.apache.org/notsoquick.html
    http://blog.ibd.com/howto/experience-installing-hbase-0-20-0-cluster-on-ubuntu-9-04-and-ec2/

    Unfortunately, I don't think I saved the actual configuration files we ended up with, but they were extremely minimal.

    ReplyDelete

  8. Any measure of performance without lots of concurrency is not an accurate benchmark of HBase or Cassandra.

    ReplyDelete

  9. Hey David,

    I would agree, but it should be able to handle a single thread just as well. Setting up a test with lots of concurrency was part of our plan, but after HBase crashed on a single thread, we basically short-circuited the performance tests.

    Besides, we weren't looking to do a very deep exhaustive benchmark of the technologies as other sites have already done, we just wanted to get a feel for which would work best for us in our situation.

    Jesse

    ReplyDelete

  10. Hi~~

    What version of Cassandra, Hbase do you test?
    and How many cluster do you use?

    thank you~~~

    ReplyDelete

  11. Hey CharSyam,

    Cassandra was version 0.7.
    HBase was version 0.20.6.
    Our tests used a single cluster of three machines each.

    Jesse

    ReplyDelete

  12. Hi Jesse,

    just to clarify the performance: 500ms per write, 1 million records: so it took approx 6 days to run the test? They seem very slow ?

    ReplyDelete

  13. Hi Mogol,

    It's in microseconds, not milliseconds.

    ReplyDelete

  15. On number 8, I am on both email lists cassandra and hbase and I get way more email on hbase(I am not on the irc channels though). ie. maybe it's hard to tell...I thought the hbase community was bigger until you mentioned the irc channels(maybe it's more of which community is using what for means of communication and both communities are the same size?)

    ReplyDelete

  16. oh, and another good reference is how many companies search for people who know hadoop vs. how many search for cassandra(if you add java, it is essentially a tie ;) )...

    http://www.indeed.com/jobtrends?q=hadoop%2C+cassandra&l=

    ReplyDelete

  17. DataStax Enterprise (DSE) includes a Cassandra-enabled Hive/Pig MapReduce client.

    http://www.datastax.com/docs/1.0/datastax_enterprise/about_hive

    DataStax Enterprise (DSE) is a commercial distribution of Apache Cassandra and Apache Hadoop developed by DataStax. DSE provides Hadoop MapReduce capabilities using CassandraFS, an HDFS-compatible storage layer inside of Cassandra. By replacing HDFS with CassandraFS, users are able to leverage their current MapReduce jobs on Cassandra’s peer-to-peer, fault-tolerant, and scalable architecture. DataStax Enterprise is also able to support dual workloads, allowing you to use the same cluster of machines for both real-time applications and data analytics without having to move the data around between systems.

    ReplyDelete

  18. I wonder how much memory they used during stress tests and performance tests. Did you checked those or have any idea about approximate usages?

    ReplyDelete

  19. Hi, I dont see any read performance metrics in your test. Is there any reason for skipping them?

    ReplyDelete

  20. Hi,
    nice comparison, we did something similar, but excluded HBase and Cassandra, because they "looked" complex at a time.

    at the end we evaluated Cassandra too, but did not evaluated HBase.

    Why we choose Apache Cassandra over Riak and Tokyo Cabinet
    http://nmmm.nu/cass.htm

    ReplyDelete

  21. Thanks for such a great comparison! When we look at HBase and Cassandra, they can look very similar. They’re both part of the NoSQL ecosystem. Although they’re capable of handling very similar workloads, there are also quite a few differences. Cassandra is designed from the ground up to handle very high, concurrent, write-intensive workloads. HBase on the other hand, has its limitations in scalability, and may require a bit more thinking to achieve the same quality of service. More at Hadoop Online Training

    ReplyDelete

  29. Il Chelsea 16-17 terzo kit dispone di una combinazione di colori tradizionale in bianco e blu. Realizzato da Adidas, la nuova terza maglia del Chelsea 2016-2017 è stato rilasciato il 29 luglio ed è impostato per funzionare come Maglia nelle Coppe.maglie calcio 2017,
    maglie calcio poco prezzo, Maglia Barcelona 2017 vendita
    Maglia Deportivo a poco prezzo shop

    ReplyDelete

  30. This house flag is Officially Licensed by the NFL and is a high quality item for hanging from your porch or as an interior wall banner.Hang your Minnesota Vikings Double Sided House Flag with our banner pole or fly it with our 5' or 6' flagpoles and adjustable flag brackets.
    house divided flags nhl
    nfl house flagsTennessee Titans house divided flags
    buy Cleveland Browns house divided flags

    ReplyDelete

  31. Quester is a Pakistan-based questioning answering website where people can ask questions and we try our best to provide them with the best answers. Anyone can ask any legit question in English or Roman Urdu and we provide answers in the same language format. www.quester.pk

    ReplyDelete

  36. best web To make yourselves available FNAF Friv Friv Games Thank you both for sharing Holy Mass with us each day Friv Games Twizl Juegos De FNAF when we were called out late at night and any other time and for your support and encouragement in every respect. Friv 2021 Twizl Juegos Yepi Juegos Twizl Thank you for sharing with me a few of your ideas regarding the possibility.

    ReplyDelete

  37. nice blog, thank you
    Danke, dafur dass Ihr jeden Tag mit uns die Hl. Messe gefeiert habt, fur all die Reparaturen im ganzen Haus Friv 10 Friv 2018 Friv 2019 fur Euere Bereitschaft Friv 10 Friv 20 Friv 2 immer zur Verfugung zu stehen Geometry Dash Twizl Twizy Twizl Danke, dass Sie Ihr Muhen um den Aufbau des Leibes Christi mit uns teilten.

    ReplyDelete

  38. Senhor Deputado Cashman, agradeço-lhe a informação. Juegos Friv Jogos Friv Friv4school A todos que partilham e trabalham sob estas mesmas convicções e princípios Jeux De Friv Juegos De Roblox Zoxy Mais uma vez, obrigada ao Parlamento por comungar da visão que informa a nova política dos consumidores Juegos Kizi Juegos Yepi Twizl Zoxy assente no mercado - a visão de um mercado de consumidores informados e capacitados que procuram e usufruem, com confiança,

    ReplyDelete

  39. Wir danken Ihnen
    Friv4school Gry Friv Juegos Friv dass Sie diese Hoffnung mit uns teilen und diesen Schritt auf dem Pilgerweg des Vertrauens Friv Juegos Friv Gry Friv mit uns gegangen sind. Juegos Friv Juegos De Friv Ihnen hiermit die gewünschten Informationen vermitteln zu können. Yepi Juegos Friv wir danken Ihnen für das Interesse an unseren Produkten und hoffe

    ReplyDelete

  42. This article is interesting and useful. Thank you for sharing. And let me share an article about health that God willing will be very useful. Thank you :)

    Obat gatal Kudis/gudik
    Walatra Gamat Emas Kapsul
    Vitamin Untuk Kesehatan Anak
    Penyebab sering mimisan
    Cara Mengatasi Cacar Air
    Cara Menghilangkan Kantung mata
    Obat Telinga Berkerak dan Berair

    ReplyDelete

  46. Generally most payday advance organizations charge around $20 to $30 per $100 of loan. To guarantee that there are no ambiguities, demand that your last payday credit sum is explained in dollars and pennies. This will enable you to realize what you have to pay, when your payday credit is expected. Payday Loans Chicago

    ReplyDelete

Subscribe to: Post Comments (Atom)