Wednesday, February 2, 2011

HBase vs Cassandra

Adku recently decided to migrate part of it's data to a NoSQL database in order to deal with increasing load on our MySQL database. We evaluated many options including MongoDB, Amazon SimpleDB, and a few others, but ultimately narrowed the options to just Cassandra and HBase. We experimented with both databases and evaluated them more deeply and ultimately decided to use Cassandra. This post explains the high level reasons why we chose Cassandra.

Disclaimer: While these decisions apply to Adku, they might not apply to your situation. Always do your own investigation and experimentation before choosing any large part of your system.
  1. Reliability
    - We wrote a stress test tool to simulate how the databases would behave under high load. We used the default minimalistic configuration for each database following their respective documentations. Our stress test inserted 1 million rows of the full "Alice in Wonderland" text (~180kb) inserted into each row done continuously with one concurrent thread into a 3 node cluster. Surprisingly, the HBase region servers actually crashed on us consistently. Cassandra never crashed once. Although we would obviously scale the cluster under high loads and crashing nodes, this is still definitely concerning and a large win for Cassandra.

  2. Performance
    - We also wrote a simulation tool to test a more realistic scenario to see how each database would respond. Similar to the above, our tool inserted 1 million rows with 2000 characters of ascii text inserted into each row done continuously with one concurrent thread into a 3 node cluster. HBase averaged 507 microseconds per write where Cassandra averaged 480 microseconds per write. We interpreted this as basically equivalent performance so there was no real winner here.

  3. Consistency
    - Consistency is not a hard requirement for our specific use case so Cassandra's eventual consistency model is fine for us. If we do end up needing consistency, Cassandra can support it using their configurable CAP model, we would just have to take a performance hit. HBase has consistency so it is technically the winner here, but since it's not a hard requirement for us, it doesn't carry much weight.

  4. Single Point of Failure
    - Hadoop's namenode which HBase depends on is a single point of failure. This means that if the namenode goes down, the entire database is unreachable. All Cassandra nodes are identical so there is no single point of failure. This is a win for Cassandra.

  5. Hot Spot Problem
    - Our relevant row keys are currently all timestamps. HBase chooses the node to store data on by row key in sorted order. Cassandra by default stores them on a random node in random order. This means that HBase will fall into a hotspot problem where one node is handling most of the write traffic. Cassandra, however, distributes the load across all nodes evenly. This is a win for Cassandra.

  6. MapReduce
    - HBase is built on top of HDFS and Hadoop. This means that MapReduce is very easy. Cassandra supports MapReduce, but doesn't support streaming MapReduce so you have to write them in Java. I'm also not sure what the relative performance is. Cassandra supports data locality so that MapReduces tasks end up processing data on the same machine as the MapReduce task so it's possible that performance is comparable, but I haven't done adequate tests. HBase is the winner here, but so far we haven't seen any drawbacks to running MapReduce on Cassandra (besides having to write in Java).

  7. Simpler, Hackable
    - Cassandra is a simpler implementation and much easier to hack. This is the same reason why we chose Tornado instead of Apache as our web server and we've actually made quite a few modifications to the Tornado web server as a result. Bugs are also much easier to debug. HBase by comparison is much more complicated and harder to debug and hack. This is a win for Cassandra and we've even already submitted patches back into open source Cassandra.

  8. Community Support
    - As of today, there are 175 users in the #cassandra channel on irc.freenode.net. HBase by comparison only has 74. Aside from IRC, it does appear that the Cassandra community is larger and more helpful than HBase. Another win for Cassandra.

94 comments:

  1. Glad that you found a solution that works for you.

    Your analysis of #4, "Single Point of Failure", contains some inaccuracies, though.

    You state, regarding HBase, "This means that when the master goes down, although the system can automatically recover, this recovery takes on the order of seconds during which your database is down."

    This simply is not true. The master is not involved at all in handling of normal client requests. All data in the cluster will continue to be available, for both reads and writes, in the event of a master failure. The cluster can continue operating without any master at all. The only operations impacted will be HBase internal operations, such as region splits.

    ReplyDelete
  2. On the Cassandra is simpler than HBase, I have to ask, by what measure?

    Is it by this measure:
    cassandra$ find test -iname \*.java | xargs wc -l | tail -1
    7011 total
    cassandra$ cd ../hbase_trunk
    hbase_trunk ryan$ find src/test -iname \*.java | xargs wc -l | tail -1
    52090 total

    Because it seems like you might be throwing the baby out with the bathwater on #7 there.

    ReplyDelete
  3. Hi Gary H, You're absolutely right. I confused the HBase master with the Hadoop namenode. I've edited the post to correct the mistake. Thanks!

    ReplyDelete
  4. Hi Ryan, We called Cassandra "simpler" because it doesn't depend on Hadoop, HDFS, and ZooKeeper which means fewer moving parts.

    ReplyDelete
  5. Interesting article - thanks. I am curious - what version of HBase did you use and how did you have the cluster configured?

    ReplyDelete
  6. Hey Robert, We used HBase 0.20.6 and we followed the configuration instructions in these two articles: hbashttp://hbase.apache.org/notsoquick.html
    http://blog.ibd.com/howto/experience-installing-hbase-0-20-0-cluster-on-ubuntu-9-04-and-ec2/

    Unfortunately, I don't think I saved the actual configuration files we ended up with, but they were extremely minimal.

    ReplyDelete
  7. Any measure of performance without lots of concurrency is not an accurate benchmark of HBase or Cassandra.

    ReplyDelete
  8. Hey David,

    I would agree, but it should be able to handle a single thread just as well. Setting up a test with lots of concurrency was part of our plan, but after HBase crashed on a single thread, we basically short-circuited the performance tests.

    Besides, we weren't looking to do a very deep exhaustive benchmark of the technologies as other sites have already done, we just wanted to get a feel for which would work best for us in our situation.

    Jesse

    ReplyDelete
  9. Hi~~

    What version of Cassandra, Hbase do you test?
    and How many cluster do you use?

    thank you~~~

    ReplyDelete
  10. Hey CharSyam,

    Cassandra was version 0.7.
    HBase was version 0.20.6.
    Our tests used a single cluster of three machines each.

    Jesse

    ReplyDelete
  11. Hi Jesse,

    just to clarify the performance: 500ms per write, 1 million records: so it took approx 6 days to run the test? They seem very slow ?

    ReplyDelete
  12. Hi Mogol,

    It's in microseconds, not milliseconds.

    ReplyDelete
  13. On number 8, I am on both email lists cassandra and hbase and I get way more email on hbase(I am not on the irc channels though). ie. maybe it's hard to tell...I thought the hbase community was bigger until you mentioned the irc channels(maybe it's more of which community is using what for means of communication and both communities are the same size?)

    ReplyDelete
  14. oh, and another good reference is how many companies search for people who know hadoop vs. how many search for cassandra(if you add java, it is essentially a tie ;) )...

    http://www.indeed.com/jobtrends?q=hadoop%2C+cassandra&l=

    ReplyDelete
  15. DataStax Enterprise (DSE) includes a Cassandra-enabled Hive/Pig MapReduce client.

    http://www.datastax.com/docs/1.0/datastax_enterprise/about_hive

    DataStax Enterprise (DSE) is a commercial distribution of Apache Cassandra and Apache Hadoop developed by DataStax. DSE provides Hadoop MapReduce capabilities using CassandraFS, an HDFS-compatible storage layer inside of Cassandra. By replacing HDFS with CassandraFS, users are able to leverage their current MapReduce jobs on Cassandra’s peer-to-peer, fault-tolerant, and scalable architecture. DataStax Enterprise is also able to support dual workloads, allowing you to use the same cluster of machines for both real-time applications and data analytics without having to move the data around between systems.

    ReplyDelete
  16. I wonder how much memory they used during stress tests and performance tests. Did you checked those or have any idea about approximate usages?

    ReplyDelete
  17. Hi, I dont see any read performance metrics in your test. Is there any reason for skipping them?

    ReplyDelete
  18. Hi,
    nice comparison, we did something similar, but excluded HBase and Cassandra, because they "looked" complex at a time.

    at the end we evaluated Cassandra too, but did not evaluated HBase.

    Why we choose Apache Cassandra over Riak and Tokyo Cabinet
    http://nmmm.nu/cass.htm

    ReplyDelete
  19. Thanks for such a great comparison! When we look at HBase and Cassandra, they can look very similar. They’re both part of the NoSQL ecosystem. Although they’re capable of handling very similar workloads, there are also quite a few differences. Cassandra is designed from the ground up to handle very high, concurrent, write-intensive workloads. HBase on the other hand, has its limitations in scalability, and may require a bit more thinking to achieve the same quality of service. More at Hadoop Online Training

    ReplyDelete
  20. Il Chelsea 16-17 terzo kit dispone di una combinazione di colori tradizionale in bianco e blu. Realizzato da Adidas, la nuova terza maglia del Chelsea 2016-2017 è stato rilasciato il 29 luglio ed è impostato per funzionare come Maglia nelle Coppe.maglie calcio 2017,
    maglie calcio poco prezzo, Maglia Barcelona 2017 vendita
    Maglia Deportivo a poco prezzo shop

    ReplyDelete
  21. This house flag is Officially Licensed by the NFL and is a high quality item for hanging from your porch or as an interior wall banner.Hang your Minnesota Vikings Double Sided House Flag with our banner pole or fly it with our 5' or 6' flagpoles and adjustable flag brackets.
    house divided flags nhl
    nfl house flagsTennessee Titans house divided flags
    buy Cleveland Browns house divided flags

    ReplyDelete
  22. Quester is a Pakistan-based questioning answering website where people can ask questions and we try our best to provide them with the best answers. Anyone can ask any legit question in English or Roman Urdu and we provide answers in the same language format. www.quester.pk

    ReplyDelete
  23. best web To make yourselves available FNAF Friv Friv Games Thank you both for sharing Holy Mass with us each day Friv Games Twizl Juegos De FNAF when we were called out late at night and any other time and for your support and encouragement in every respect. Friv 2021 Twizl Juegos Yepi Juegos Twizl Thank you for sharing with me a few of your ideas regarding the possibility.

    ReplyDelete
  24. nice blog, thank you
    Danke, dafur dass Ihr jeden Tag mit uns die Hl. Messe gefeiert habt, fur all die Reparaturen im ganzen Haus Friv 10 Friv 2018 Friv 2019 fur Euere Bereitschaft Friv 10 Friv 20 Friv 2 immer zur Verfugung zu stehen Geometry Dash Twizl Twizy Twizl Danke, dass Sie Ihr Muhen um den Aufbau des Leibes Christi mit uns teilten.

    ReplyDelete
  25. Senhor Deputado Cashman, agradeço-lhe a informação. Juegos Friv Jogos Friv Friv4school A todos que partilham e trabalham sob estas mesmas convicções e princípios Jeux De Friv Juegos De Roblox Zoxy Mais uma vez, obrigada ao Parlamento por comungar da visão que informa a nova política dos consumidores Juegos Kizi Juegos Yepi Twizl Zoxy assente no mercado - a visão de um mercado de consumidores informados e capacitados que procuram e usufruem, com confiança,

    ReplyDelete
  26. Wir danken Ihnen
    Friv4school Gry Friv Juegos Friv dass Sie diese Hoffnung mit uns teilen und diesen Schritt auf dem Pilgerweg des Vertrauens Friv Juegos Friv Gry Friv mit uns gegangen sind. Juegos Friv Juegos De Friv Ihnen hiermit die gewünschten Informationen vermitteln zu können. Yepi Juegos Friv wir danken Ihnen für das Interesse an unseren Produkten und hoffe

    ReplyDelete
  27. This article is interesting and useful. Thank you for sharing. And let me share an article about health that God willing will be very useful. Thank you :)

    Obat gatal Kudis/gudik
    Walatra Gamat Emas Kapsul
    Vitamin Untuk Kesehatan Anak
    Penyebab sering mimisan
    Cara Mengatasi Cacar Air
    Cara Menghilangkan Kantung mata
    Obat Telinga Berkerak dan Berair

    ReplyDelete
  28. Generally most payday advance organizations charge around $20 to $30 per $100 of loan. To guarantee that there are no ambiguities, demand that your last payday credit sum is explained in dollars and pennies. This will enable you to realize what you have to pay, when your payday credit is expected. Payday Loans Chicago

    ReplyDelete
  29. يسعي الكثير من الباحثين الي الحصول علي خدمة الترجمة البحثية من قبل خبراء في مجال الترجمة من خلال موقع مكتبتك التي يقدم العديد من الانواع المختلفة للخدمات

    ReplyDelete
  30. يسعي الكثير من الباحثين الي التعامل مع موقع مكتبتك لجودة الاعمال التي يتم القيام بها علي درجة عالية من الجودة والدقة منها خدمة اعدراد اطار نظري للبحث العلمي بجودة عالية

    ReplyDelete
  31. يهتم فريق عمل موقع مكتبتك بتقديم أفضل وأدق ترجمة طبية يمكن الحصول عليها وذلك من خلال المقارنة مع المواقع الالكترونية الاخري لجذب انتباه الباحث العلمي من خلال عرض ترجمة نصوص علمية في كافة المجالات العلمية والطبية

    ReplyDelete
  32. I am really happy to read this blog post, you have many information put it on this article, am really suggest to my all dear friend to visit this blog post and collect this useful and valuable information, keep sharing like this type of blog post, thank you so much for read my comment, if any one searching top 5 website designing company in India please come on my website we are doing for you any type of website design for your, business grow up your work grow up your money.
    Top 5 Website Designing Company in India

    ReplyDelete
  33. Am visit of many blog but not get valuable information, but after seen your blog provide it me a lot of valuable and useful information I really like to thank you so much for share this information with us, if are inserted in grow up your life style and also grow up your looking so please come on my website am put it on many information so and visit again time thank you so much for read my comment I hope your also like my comment and you also go on my web page.
    Lifestyle Magazine India

    ReplyDelete
  34. That was such an awesome content to read and going through it.Thanks for such a good information.our product related for servo voltage stabilizer and transformer.
    Servo Stabilizer Manufacturer in india

    What is Servo Voltage Stabilizer?

    Distribution Transformers Manufacturer in india

    What is Step Down Transformer?

    ReplyDelete
  35. Given article is very helpful and very useful for my admin, and pardon me permission to share articles here hopefully helped :

    Cara Mengatasi Perih Di Ulu Hati Secara Alami
    Cara Mengobati Sirosis Hati Secara Alami

    ReplyDelete
  36. May be useful for all, helpful article once and pardon me permission to share also here :

    Pengobatan Maag Kronis Secara Tradisional

    ReplyDelete
  37. http://www.duasinislam.com/pari-ka-amal/pari-ko-pane-ka-amal/

    ReplyDelete
  38. Thanks for sharing your wealthy information. This is one of the excellent posts which I have seen. I go through your all of your blog, but this blog is the best one.
    Top 5 Web Designing Company In India | SEO Services In India

    ReplyDelete
  39. See the latest customer reviews, feedbacks, and complaints about our country top packers and movers. Like Agarwal Packers and Movers. How was the shifting service of Agarwal Packers and Movers? See now Agarwal Packers and Movers Customer Reviews.

    CLICK HERE

    ReplyDelete
  40. Wow, amazing blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your website is fantastic, let alone the content!

    3d animation Company
    Best Chatbot Development Company
    Mobile app development in Coimbatore

    ReplyDelete
  41. Thank you so much for share this type of articles keeps the users interest in this article, I really like to read this awesome blog.
    Transformer Manufacturers In Pune | Transformer Manufacturers In Mumbai | Transformer Manufacturers In India

    ReplyDelete
  42. I am very interested in this post, everyone should know it more, wish you success. The author of this post is admirable, it brings a lot of emotion to everyone reading it, let's share it more widely.
    free friv Games 2019, Jogos de Friv, friv for school free Games, cá koi mini

    ReplyDelete
  43. A number of cranes of various sizes are used to transport furniture properly.
    It has many cars that are equipped to do the job to the fullest.
    Furniture is well packaged before moving from one place to another.
    When the furniture is delivered to the desired location, the staff will install the furniture with each piece in place.شركة نقل عفش
    شركة نقل اثاث من الرياض الى جدة

    شركة نقل عفش من الرياض الى جدة

    ReplyDelete
  44. Thanks for taking the time to discuss and share with us. I wish more people would read this. We have expert developers with over 12+ years of experience. If you have any requirement regarding Android & iOS mobile app development then you can contact us at +1-877-659-9068.

    ReplyDelete