Friday, March 16, 2012

Hanborq Improved Hadoop MapReduce

Major Features:
1. Worker Pool
Does not spawn new JVM processes for each job/task, but instead start these slot/worker processes at initialization phase and keep them running constantly.
2. Sort Avoidance.
Many aggregation job need not sort.
A Hanborq optimized Hadoop Distribution, especially with high performance of MapReduce. It's the core part of HDH (Hanborq Distribution with Hadoop for Big Data Engineering).
Here is our presentation: Hanborq Optimizations on Hadoop MapReduce

HDH (Hanborq Distribution with Hadoop)

Hanborq, a start-up team focuses on Cloud & BigData products and businesses, delivers a series of software products for Big Data Engineering, including a optimized Hadoop Distribution.
HDH delivers a series of improvements on Hadoop Core, and Hadoop-based tools and applications for putting Hadoop to work solving Big Data problems in production. HDH is ideal for enterprises seeking an integrated, fast, simple, and robust Hadoop Distribution. In particular, if you think your MapReduce jobs are slow and low performing, the HDH may be you choice.

Hanborq optimized Hadoop

It is a open source distribution, to make Hadoop FastSimple and Robust.
- Fast: High performance, fast MapReduce job execution, low latency.
- Simple: Easy to use and develop BigData applications on Hadoop.
- Robust: Make hadoop more stable.

MapReduce Benchmarks

The Testbed: 5 node cluster (4 slaves), 8 map slots and 2 reduce slots per node.
1. MapReduce Runtime Environment Improvement
In order to reduce job latency, HDH implements Distributed Worker Pool like Google Tenzing. HDH MapReduce framework does not spawn new JVM processes for each job/task, but instead keep the slot processes running constantly. Additionally, there are many other improvements at this aspect.
bin/hadoop jar hadoop-examples-0.20.2-hdh3u3.jar sleep -m 32 -r 4 -mt 1 -rt 1
bin/hadoop jar hadoop-examples-0.20.2-hdh3u3.jar sleep -m 96 -r 4 -mt 1 -rt 1
HDH MapReduce Runtime Job/Task Latency
2. MapReduce Processing Engine Improvement
Many improvements are applied on Hadoop MapReduce Processing engine, such as shuffle, sort-avoidance, etc. HDH MapReduce Processing Engine Benchmark
Please refer to the page MapReduce Benchmarks for detail.



- Fast job launching: such as the time of job lunching drop from 20 seconds to 1 second.
- Low latency: not only job setup, job cleanup, but also data shuffle, etc.
- High performance shuffle: low overhead of CPU, network, memory, disk, etc.
- Sort avoidance: some case of jobs need not sorting, which result in too many unnecessary system overhead and long latency.
... and more and continuous ...

How to build?

$ cd cloudera/maven-packaging  
$ mvn -Dmaven.test.skip=true -DskipTests=true clean package  
Then use this package: build/hadoop-{main-version}-hdh{hdh-version}, for example: build/hadoop-0.20.2-hdh3u2


The API, configuration, scripts are all back-compatible with Apache Hadoop and Cloudera Hadoop(CDH). The user and developer need not to study new, except new features.

Innovations and Inspirations

The open source community and our real enterprise businesses are the strong source of our continuous innovations. Google, the great father of MapReduce, GFS, etc., always outputs papers and experiences that bring us inspirations, such as:
MapReduce: Simplified Data Processing on Large Clusters
MapReduce: A Flexible Data Processing Tool
Tenzing: A SQL Implementation On The MapReduce Framework
Dremel: Interactive Analysis of Web-Scale Datasets
... and more and more ...

Open Source License

All Hanborq offered code is licensed under the Apache License, Version 2.0. And others follow the original license announcement.


  1. Nice blog i like this post on hadoop. i am looking for such information long time & finally i got it from this post,Thanks for sharing this great information.
    Hadoop Training in Hyderabad

  2. Thanks you for sharing the unique content. you have done a great job. I appreciate your effort and I hope that you will get more positive comments from the web users.

    Hadoop training in chennai

  3. thank you for sharing such a nice and interesting blog with us. hope it might be much useful for us. keep on updating...
    SEO Company In Chennai

  4. Superb explanation & it's too clear to understand the concept as well, keep sharing admin with some updated information with right examples.Keep update more posts.

    SEO Company in India

  5. This is extremely helpful info!! Very good work. It is very interesting to learn and easy to understood. Thank you for giving information. Please let us know and more information get post to link.

    Informatica Training in Chennai

  6. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much
    SAP Training in Chennai
    SAP Basis Training in Chennai
    SAP SD Training in Chennai
    SAP FICO Training in Chennai

  7. This is a great article, I have been always to read something with specific tips! I will have to work on the time for scheduling my learning.
    Hadoop Training in Chennai

  8. Really i like this blog and i got lot of information's from your blog.And thanks for sharing!!!!
    Human resources management software
    cctv camera installation in Chennai
    RFID Solutions

  9. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Android Training in Chennai
    Ios Training in Chennai

  10. Your blog is very unique and interesting. It makes reader to come back and visit again.

    Selenium Training in Chennai

  11. i have really enjoyed sharing your website. thank you so much for your sharing this document. this document more useful and improve our knowledge.
    salesforce training in chennai

  12. A great content and very much useful to the visitors. Looking for more updates in future.

    PYTHON Training in Chennai

  13. Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. Thanks for sharing
    SAP Training in Chennai
    SAP ABAP Training in Chennai


  14. It is really very excellent,I find all articles was amazing.Awesome way to get exert tips from everyone,not only i like that post all peoples like that post.Because of all given information was wonderful and it's very helpful for me.
    Hadoop Training in Chennai
    Data Warehousing Training in Chennai
    Informatica Training in Chennai
    SAS Training in Chennai

  15. Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. Thanks for sharing....

    Carwash in omr
    usedcars in omr
    automotors in omr
    car accessories in omr
    secondhand car in omr

  16. samsung galaxy cases
    It's fascinating that a significant number of the bloggers to cleared up a couple of things for me and in addition giving.Most of thoughts can be decent content.The individuals to give them a decent shake to get your point and over the charge.

  17. Accountants Hove
    when you post the new artical iam excited to read more artical on this site

  18. To maximize impact of RPA, identify impactful processes. These processes tend to be Impacting both cost and revenues , High volume , With low fault tolerance , Error prone , Speed-sensitive , Requiring irregular labor , Distributed processes that require coordinated efforts of multiple departments ,Select processes that can be easily automated with RPA. Such processes tend to be
    Rules based
    Not on the roadmap for new systems
    Robotic Process Automation RPA training in chennai

  19. All the latest updates from the Python Automationminds team. Python Automationminds lets you program in Python, in your browser. No need to install any software, just start coding straight away. There's a fully-functional web-based console and a programmer's text-editor
    Phyton training in Chennai

  20. Excellent and very cool idea and the subject at the top of magnificence and I am happy to this post..Interesting post! Thanks for writing it. What's wrong with this kind of post exactly? It follows your previous guideline for post length as well as clarity..
    CRM Software in india
    CRM Software in Chennai

  21. I do trust all of the concepts you’ve presented on your post. They’re really convincing and will definitely work. Still, the posts are too brief for newbies. May you please extend them a little from subsequent time?Also, I’ve shared your website in my social networks.
    Mobile App Development Company In Chennai
    Android App Development Company In Chennai
    Android Application Development Company In Chennai
    Custom Web Application Development Company In Chennai

  22. شركة اللمسة الأخيرة تقدم لك الحل الأمثل فلا حشرات بعد اليوم ولن تعود مرة أخرى. فنحن نستخدم أفضل المبيدات العالمية الفعالة صديقة البيئة التي لا تترك رائحة ولا سيوثر على صحة الأنسان ويقوم باستخدامها عمال مدربون يقومون برش المبيدات بشكل علمي مما يضمن لك الراحة التامة نرجو التواصل على هذا الرقم 0580002467
    شركة رش مبيدات بأبها
    شركة مكافحة حشرات بأبها
    شركة مكافحة النمل الابيض بأبها
    شركة رش مبيدات بخميس مشيط
    شركة مكافحة حشرات بخميس مشيط
    شركة مكافحة النمل الابيض بخميس مشيط
    شركة رش مبيدات بالقصيم
    شركة مكافحة حشرات بالقصيم
    شركة مكافحة حشرات بجازان
    شركة رش مبيدات بجازان

  23. Your blog is very nice! Thank you for this wonderful information . i like very much for this article. and it will give the more information from this blog. I also waiting for more than one information this is to get some more processes are available.Thanks for this nice article.

  24. A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

  25. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

  26. This comment has been removed by the author.