Friday, May 27, 2011

Giving back - Extending MRJob

If you’ve ever used MRJob to run a MapReduce and wanted to debug a problem, you’ll soon realize that attaching a debugger to your mappers and reducers is very difficult. We’ve solved this by building a local, inline job runner that makes it as easy to debug your MapReduce as if it were a standalone Python script.

Over the last few weeks, Adku has been playing around with a cool new project named MRJob, a python package that streamlines the development of MapReduce jobs. One of the core features of MRJob is that you can write your MapReduce job once (in Python!) and run it pretty much anywhere. Out of the box, MRJob comes with support for launching your jobs onto a Hadoop cluster. It also comes with built-in support for launching jobs directly on Amazon's ElasticMapReduce (EMR) service. Why is this exciting you ask? Well, Amazon makes it simple to launch arbitrarily large/small Hadoop clusters in a matter of minutes. If you need a tiny cluster to test that your job works remotely, you can run your cluster on a single “m1.small” instance. If you need a huge Hadoop cluster to crunch gigabytes or even terabytes worth of data, you can spin up a cluster with dozens of “m1.large” instances.

Before spinning up hundreds of machines to run your mind-blowing MapReduce job, you'll probably want to test that your job works before blowing through your IT budget. Nothing sucks more than spinning up a huge cluster, waiting for Amazon to provision all your machines, then find out that you had a typo in your job. Because Amazon rounds up on instance-hours used, your 5 minute jaunt to nowhere just cost you “$/instance-hour * # of instances * 1 hour” Luckily, MRJob provides ways for you to test your job before launching on EMR.

Out of the box, MRJob comes with support for a local job runner that acts like a mini-Hadoop cluster with a single mapper and a single reducer. The local runner works by first trying to emulate the environment you would see in Hadoop. Then it tries to run the equivalent of “python mapper -> sorter -> python reducer” Because the local job runner tries to clone the remote environment, this is for testing that you’ve uploaded all job dependencies. The local job runner tries to emulate Hadoop even more by using pipes to Hadoop streaming. Unfortunately, because the local job runner works by spinning up subprocesses, your favorite debugger will not be able to attach to your map/reduce phases.

Over the last few weeks, we've struggled with debugging our local MapReduce jobs. Raising exceptions in the map/reduce phases feels gross. Emitting counters with entire error messages felt even worse. If you’ve ever tried to debug a mapreduce job in a debugger or an IDE, you’ll know how difficult it is to get something as simple as a traceback. Really, what we want do is attach a debugger to our map/reduce phase to see exactly what was going on. After some poking and prodding at MRJob's internals, we figured out a way to run mappers and reducers in the same process as the launcher script. By running your job with “--runner inline” you can now attach a debugger to your job and observe pretty much everything! “--runner inline” is designed to be used in conjunction with “--runner local” It is not meant to replace the default run mode (local) as we do NOT faithfully replicate the file system you would see in Hadoop. This runner is purely for development/debugging purposes.

Where can you get this patch you ask? Well, we love giving back to the community so we sent our patch upstream a few days ago. Today, I’m happy to report that the maintainers over at Yelp accepted our changeset and has released as a part of mrjob 0.2.6!

PyPi registration - http://pypi.python.org/pypi/mrjob

Homepage - https://github.com/Yelp/mrjob
Documentation - http://packages.python.org/mrjob/

-Matt

86 comments:

  1. You have certainly explained that cluster is the process of examining big data to uncover hidden patterns, unknown correlations and other useful information that can be used to make better decisions..The big data analytics is the major part to be understood regarding training program. Via your quality content i get to know about that in deep.Thanks for sharing this here.

    Hadoop course in Tambaram

    ReplyDelete
  2. You have clearly cleared up that package is the procedure of looking at gigantic information to reveal secured outlines, dull associations and other essential data that can be utilized to upgrade decisions..The colossal information examination is the real part to be comprehended with respect to get prepared undertaking. By strategy for your quality substance i end up being more acquainted with about that in deep.Thanks for sharing this here.
    Oracle DBA Training in Chennai

    ReplyDelete
  3. Thanks for your informative guide on SQL technology. As we all know, SQL is most used database tool and your article offer clear insight about the technology. Your blog is recommended for freshers and experienced ones looking for ideal blog to learn SQL technology.
    .Net Training in Chennai
    PHP Training in Chennai
    Big Data Training in Chennai

    ReplyDelete
  4. The expansion of internet and other business intelligence leads to large volume of data. Industries are looking for talented professionals to maintain and process huge volume of data with latest tools available in the market. Taking Hadoop Training in Chennai | Big Data Training in Chennai will ensure better career prospects for talented professionals.

    ReplyDelete
  5. Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing.
    Regards,
    SAP training institute in chennai|Best SAP training in chennai|SAP Training in Chennai|SAP Course in Chennai

    ReplyDelete
  6. This was worth reading article, java is no doubt the most powerful programming languages yet more demanding as well. J2EE Training in Chennai | JAVA Training in Chennai

    ReplyDelete
  7. Thanks for your informative article on software testing. Your post helped me to understand the future and career prospects in software testing. Keep on updating your blog with such awesome article. testing training in Chennai| Software testing training in Chennai

    ReplyDelete
  8. Thanks for your article on Android technology. Android is an open source platform that allows developers to create stunning website loaded with various advanced features and functionalities. Android Training in Chennai

    ReplyDelete
  9. I gathered useful information on this point . Thank you posting relative information and its now becoming easier to complete this assignment
    mahjong |geometry dash | hulk|agario| kizi|sniper games| minecraft|halloween | pacman| games

    ReplyDelete
  10. Excellent post on iOS mobile apps development!!! The future of mobile application development is on positive note. You can make most it by having in-depth knowledge on mobile application development platform and other stunning features. iOS Training in Chennai | iOS Training Institutes in Chennai

    ReplyDelete
  11. The content provided here is vital in increasing one's knowledge regarding hadoop, the way you have presented here is simply awesome. Thanks for sharing this. The uniqueness I see in your content made me to comment on this. Keep sharing article like this. Thanks :)

    Hadoop Training Chennai | Big Data Training in Chennai | hadoop course in Chennai

    ReplyDelete
  12. The one and only Operating system that is used widely by many around the world. Thanks for providing some useful information regarding the OS. This content was very helpful to me. Thanks for sharing. Keep blogging article like this.


    Android training in chennai | Android course in chennai

    ReplyDelete
  13. Dear admin, The way you have explained the concept is mezmerizing. Thank you so much for sharing tis worth able content with us. The concept taken here will be useful for my future programs and i will surely implement them in my study. Keep blogging article like this.


    Best JAVA Training institute in Chennai | Best JAVA Training in Chennai | Salesforce training in chennai

    ReplyDelete
  14. The quality of the content is absolutely great and awesome where you have explained clearly and the words are really awesome just cool blog very much useful information. I would like inform that IICT is awarded as the best iOS Training Institute in Chennai Managed by real time Experts who have around 20+ years of experience.

    ReplyDelete
  15. Hi Admin, I went through your article and it’s totally awesome. You can consider including RSS feed for easy content sharing, So that you can drive huge traffic to your blog. Hadoop Training in Chennai | Big Data Training in Chennai

    ReplyDelete
  16. Your blog is awesome..You have clearly explained about it ...Its very useful for me to know about new things..Keep on blogging..
    Software testing training in chennai

    ReplyDelete
  17. Your blog is awesome..You have clearly explained about it ...Its very useful for me to know about new things..Keep on blogging..
    IOS training in chennai

    ReplyDelete
  18. I am following your blog from the beginning, it was so distinct & I had a chance to collect conglomeration of information that helps me a lot to improvise myself.
    Regards,
    CCNA Training in Chennai | CCNA Training Institute in Chennai | Best CCNA Training in Chennai

    ReplyDelete
  19. Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up

    - Mortal Kombat XL
    - Atari Breakout
    - Dragon Ball Z Games

    ReplyDelete
  20. Very nice post e commerce development and also website designs..:) ecommerce web designing

    ReplyDelete
  21. Thanks for your post; selenium is most trusted automation tool to validate web application and browser. This tool provides precise and complete information about a software application or environment. Selenium Training in Chennai | Selenium Course in Chennai | Best Selenium training institute in Chennai

    ReplyDelete
  22. I must appreciate the way you have expressed your feelingsthrough your blog!. Click here to play
    shooting games | dog games | fighting games | mickey mouse games

    ReplyDelete
  23. Yes Offcourse cluster in and as Big data Mavjob..
    SEO training in hyderabad by experts in digital markeing And by prosessional experts in seo.All the training by placement and also guide by the professionals.SEO training in hyderabad

    ReplyDelete
  24. nice..

    SEO training in hyderabad by experts in digital markeing And by prosessional experts in seo.All the training by placement and also guide by the professionals.SEO training in hyderabad

    ReplyDelete
  25. This blog is so nice to me. I will continue to come here again and again. Visit my link as well. Good luck
    obat aborsi
    cara menggugurkan kandungan
    obat telat datang bulan
    obat penggugur kandungan

    ReplyDelete
  26. The war between humans, orcs and elves continues earn to die . Lead your race through a series of epic battles, using your crossbow to fend off foes and sending out units to destroy castleshappy wheels . Researching and upgrading wisely will be crucial to your success! There are 5 ages total and each one will bring you new units to train to fight in the war for you cause.
    earn to die 2
    Whatever you do, don’t neglect your home base because you cannot repair it and once it is destroyed, you lose! Age of War is the first game of the series and really sets the tone for the Age of War games . Also try out the Age of Defense series as it is pretty similar.
    In this game, you start at the cavern men’s age, then evolvetank trouble ! There is a total of 5 ages, each with its units and turrets. Take control of 16 different units and 15 different turrets to defend your base and destroy your enemy.
    The goal of the game also differs depending on the level. In most levels the goal is to reach a finish line or to collect tokens. Many levels feature alternate or nonexistent goals for the player.

    ReplyDelete
  27. A good blog. Thanks for sharing the information. It is very useful for my future. keep sharing
    duck life 3 | Slither io |Red Ball 3 |

    ReplyDelete
  28. nice post and site, good work! This article is well written and quite informative.
    salesforce training in hyderabad

    ReplyDelete
  29. Helpful Information
    Freshers jobs way provide freshers jobs ,walkins,direct recruitment,openings for freshers and experienced and recruiting experienced and freshers Walkins

    ReplyDelete
  30. useful information

    Online recruitment for bank jobs and government jobs and you can get Notification and application to apply online for bank jobs and govt jobs

    ReplyDelete
  31. Useful information:
    General Awareness and Current Affairs with Latest Current Affairs Quiz, GK Updates, General Knowledge Last 6 Months Current Affairs, Placement Papers, Model Papers, Interview Questions @ examvalue.com

    ReplyDelete
  32. Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    Web designing Course in Chennai | Hadoop Training in Chennai

    ReplyDelete
  33. useful information

    Online recruitment for bank jobs and government jobs and you can get Notification and application to apply online for bank jobs and govt jobs

    ReplyDelete
  34. General Awareness and Current Affairs with Latest Current Affairs Quiz, GK Updates, General Knowledge Last 6 Months Current Affairs, Placement Papers, Model Papers, Interview Questions @ examvalue.com

    ReplyDelete
  35. Wonderful post. I like your blog. Thanks for sharing.

    ppc training in chennai

    ReplyDelete
  36. Excellent post. Keep sharing such a valuable post.

    wordpress Training in Chennai

    ReplyDelete
  37. I have read your blog its very attractive and impressive. I like it your blog.

    Java Training in Chennai Java Training in Chennai | Core Java Training in Chennai Java Training in Chennai

    Online Java Training Java Online Training | Java 8 online training | JavaEE Training in Chennai

    ReplyDelete
  38. Yeoman Training JavaScript Training in Chennai

    Angular 2 Training in Chennai Angular 2 Training in Chennai

    HTML5 Training in Chennai

    JQuery Training in Chennai JQuery Training in Chennai

    D3 Training in CHennai D3 Training ReactJS Training in CHennai ReactJS Training

    Gulp TrainingGulp Training CommonJS Training in Chennai CommonJS Training

    ReplyDelete
  39. Hibernate Training Institutes in ChennaiHibernate Training Institutes in Chennai Hibernate Online Training Hibernate Online Training Hibernate Training in Chennai Hibernate Training in Chennai Java Online Training Java Online Training

    ReplyDelete
  40. I have read your blog its very attractive and impressive. I like it your blog.

    Java Training in Chennai Java Training in Chennai | Core Java Training in Chennai Java Training in Chennai

    Online Java Training Java Online Training | Java 8 online training | JavaEE Training in Chennai

    ReplyDelete
  41. I have read your blog its very attractive and impressive. I like it your blog.
    oracle training in chennai

    ReplyDelete
  42. I have read your blog its very attractive and impressive. I like it your blog.
    sas Training in Chennai

    ReplyDelete
  43. Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    Android training in Chennai | Android course in Chennai

    ReplyDelete
  44. Thanks for your sharing. Your article is very useful, it gives me more understanding. Please allow me to share a few:
    Unblocked Games - Score Hero - Photo Grid - baixar facebook

    ReplyDelete
  45. This blog is so nice to me. I will continue to come here again and again. Visit my link as well. Good luck
    http://www.jualobataborsiherbal.com/ obat aborsi
    http://caramenggugurkankandungan.info/ cara menggugurkan kandungan
    http://www.jualobataborsiherbal.com/cara-menggugurkan-kandungan/ cara menggugurkan kandungan
    http://obataborsi59.com/ obat aborsi
    http://obataborsi59.com/cara-menggugurkan-kandungan-dengan-cepat-dan-aman/ cara menggugurkan kandungan
    http://obattelatdatangbulan.info/ obat telat datang bulan
    http://klinikobataborsi.com/ jual obat aborsi
    http://jualobatpenggugurkandungan.net/ obat penggugur kandungan
    http://tandatandakehamilan.net/ tanda tanda kehamilan

    ReplyDelete
  46. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  47. your article is more useful and informative.. thanks to share this... keep rocking..

    Android Training in chennai | Android course in chennai | IOS Training in chennai

    ReplyDelete
  48. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Great work.thanks
    Juegos Yepi 2
    Kizi2
    Kongregate Game
    Money Movers 2
    Parking Mania
    Red Ball Games
    Super Fighter 2
    Tanki Online Games
    Truck Driving Game

    ReplyDelete
  49. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete

  50. Thank you both for sharing Holy Mass with us each day FNAF Sister Location Juegos Friv Friv 2018 to make yourselves available Friv Twizl Jeux De Twizl Juegos De FNAF Sister Location when we were called out late at night and any other time and for your support and encouragement in every respect. Juegos Friv 2021 Juegos Twizl Juegos Yepi 2017 Juegos De Twizl

    ReplyDelete
  51. nice blog, thank you
    Danke, dafur dass Ihr jeden Tag mit uns die Hl. Messe gefeiert habt, fur all die Reparaturen im ganzen Haus Friv 10 Friv 2018 Friv 2019 fur Euere Bereitschaft Friv 10 Friv 20 Friv 2 immer zur Verfugung zu stehen Geometry Dash Twizl Twizy Twizl Danke, dass Sie Ihr Muhen um den Aufbau des Leibes Christi mit uns teilten.

    ReplyDelete
  52. A todos que partilham e trabalham sob estas mesmas convicções e princípios Friv Friv 360 Friv4school 2020 Senhor Deputado Cashman, agradeço-lhe a informação. Jeux De Friv 2018 Juegos De Roblox Juegos De Zoxy Mais uma vez, obrigada ao Parlamento por comungar da visão que informa a nova política dos consumidores Juegos Kizi 2017 Juegos Yepi 2017 Twizl 3 Zoxy 2 assente no mercado - a visão de um mercado de consumidores informados e capacitados que procuram e usufruem, com confiança,

    ReplyDelete
  53. Wir danken Ihnen
    Friv4school Gry Friv Juegos Friv dass Sie diese Hoffnung mit uns teilen und diesen Schritt auf dem Pilgerweg des Vertrauens Friv Juegos Friv Gry Friv mit uns gegangen sind. Juegos Friv Juegos De Friv Ihnen hiermit die gewünschten Informationen vermitteln zu können. Yepi Juegos Friv wir danken Ihnen für das Interesse an unseren Produkten und hoffe

    ReplyDelete
  54. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  55. Gracias por compartir sus ideas con nosotros. Friv Games Gry Friv gracias por la detallada información. Juegos Friv Gry Friv Friv Gracias por vuestras reacciones a las transmisiones que os han hecho. Friv Games Juegos Friv Acogemos con agrado los comentarios de los lectores.

    ReplyDelete
  56. your article is more useful and informative.. thanks to share this... keep rocking..
    PHP Training Institute in Chennai

    ReplyDelete