Friday, March 16, 2012

Hanborq Improved Hadoop MapReduce

Major Features:
1. Worker Pool
Does not spawn new JVM processes for each job/task, but instead start these slot/worker processes at initialization phase and keep them running constantly.
2. Sort Avoidance.
Many aggregation job need not sort.
A Hanborq optimized Hadoop Distribution, especially with high performance of MapReduce. It's the core part of HDH (Hanborq Distribution with Hadoop for Big Data Engineering).
Here is our presentation: Hanborq Optimizations on Hadoop MapReduce

HDH (Hanborq Distribution with Hadoop)

Hanborq, a start-up team focuses on Cloud & BigData products and businesses, delivers a series of software products for Big Data Engineering, including a optimized Hadoop Distribution.
HDH delivers a series of improvements on Hadoop Core, and Hadoop-based tools and applications for putting Hadoop to work solving Big Data problems in production. HDH is ideal for enterprises seeking an integrated, fast, simple, and robust Hadoop Distribution. In particular, if you think your MapReduce jobs are slow and low performing, the HDH may be you choice.

Hanborq optimized Hadoop

It is a open source distribution, to make Hadoop FastSimple and Robust.
- Fast: High performance, fast MapReduce job execution, low latency.
- Simple: Easy to use and develop BigData applications on Hadoop.
- Robust: Make hadoop more stable.

MapReduce Benchmarks

The Testbed: 5 node cluster (4 slaves), 8 map slots and 2 reduce slots per node.
1. MapReduce Runtime Environment Improvement
In order to reduce job latency, HDH implements Distributed Worker Pool like Google Tenzing. HDH MapReduce framework does not spawn new JVM processes for each job/task, but instead keep the slot processes running constantly. Additionally, there are many other improvements at this aspect.
bin/hadoop jar hadoop-examples-0.20.2-hdh3u3.jar sleep -m 32 -r 4 -mt 1 -rt 1
bin/hadoop jar hadoop-examples-0.20.2-hdh3u3.jar sleep -m 96 -r 4 -mt 1 -rt 1
HDH MapReduce Runtime Job/Task Latency
2. MapReduce Processing Engine Improvement
Many improvements are applied on Hadoop MapReduce Processing engine, such as shuffle, sort-avoidance, etc. HDH MapReduce Processing Engine Benchmark
Please refer to the page MapReduce Benchmarks for detail.



- Fast job launching: such as the time of job lunching drop from 20 seconds to 1 second.
- Low latency: not only job setup, job cleanup, but also data shuffle, etc.
- High performance shuffle: low overhead of CPU, network, memory, disk, etc.
- Sort avoidance: some case of jobs need not sorting, which result in too many unnecessary system overhead and long latency.
... and more and continuous ...

How to build?

$ cd cloudera/maven-packaging  
$ mvn -Dmaven.test.skip=true -DskipTests=true clean package  
Then use this package: build/hadoop-{main-version}-hdh{hdh-version}, for example: build/hadoop-0.20.2-hdh3u2


The API, configuration, scripts are all back-compatible with Apache Hadoop and Cloudera Hadoop(CDH). The user and developer need not to study new, except new features.

Innovations and Inspirations

The open source community and our real enterprise businesses are the strong source of our continuous innovations. Google, the great father of MapReduce, GFS, etc., always outputs papers and experiences that bring us inspirations, such as:
MapReduce: Simplified Data Processing on Large Clusters
MapReduce: A Flexible Data Processing Tool
Tenzing: A SQL Implementation On The MapReduce Framework
Dremel: Interactive Analysis of Web-Scale Datasets
... and more and more ...

Open Source License

All Hanborq offered code is licensed under the Apache License, Version 2.0. And others follow the original license announcement.


  1. Nice blog i like this post on hadoop. i am looking for such information long time & finally i got it from this post,Thanks for sharing this great information.
    Hadoop Training in Hyderabad

  2. Thanks you for sharing the unique content. you have done a great job. I appreciate your effort and I hope that you will get more positive comments from the web users.

    Hadoop training in chennai

  3. thank you for sharing such a nice and interesting blog with us. hope it might be much useful for us. keep on updating...
    SEO Company In Chennai

  4. Superb explanation & it's too clear to understand the concept as well, keep sharing admin with some updated information with right examples.Keep update more posts.

    SEO Company in India

  5. This is extremely helpful info!! Very good work. It is very interesting to learn and easy to understood. Thank you for giving information. Please let us know and more information get post to link.

    Informatica Training in Chennai

  6. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much
    SAP Training in Chennai
    SAP Basis Training in Chennai
    SAP SD Training in Chennai
    SAP FICO Training in Chennai

  7. This is a great article, I have been always to read something with specific tips! I will have to work on the time for scheduling my learning.
    Hadoop Training in Chennai

  8. Really i like this blog and i got lot of information's from your blog.And thanks for sharing!!!!
    Human resources management software
    cctv camera installation in Chennai
    RFID Solutions

  9. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Android Training in Chennai
    Ios Training in Chennai

  10. Your blog is very unique and interesting. It makes reader to come back and visit again.

    Selenium Training in Chennai

  11. i have really enjoyed sharing your website. thank you so much for your sharing this document. this document more useful and improve our knowledge.
    salesforce training in chennai