All about hadoop Balancer.

Hadoop Data Balancing

Hadoop Balancer:

This is tool provided to balance the disk uses throughout the Hadoop cluster. I may happen sometime that some of the nodes in the cluster becomes over utilized or underutilized, which occurs due to addition of new nodes where newly added nodes may be underutilized or if there are less number of nodes result in overutilization. We can run balancer from more than 1 machine in the cluster to increase the speed of balancing but it will increase bandwidth uses to very high.

This tool requires administrator right on the Hadoop cluster to run.

Syntax of the balancer:

bin/start-balancer.sh [-threshold <threshold>]

Where start-balancer.sh files resides in the bin directory of the Hadoop folder. And the threshold is the parameter which decides target of balance, this lies in fraction between 0,1 the default value is 10% if nothing is passed as the threshold value.

This process does the transferring of blocks between the nodes resulting network activity and if a production cluster must be used cautiously, as it result in some block missing error or late reply from the cluster.

This process can be stopped any time if required using following command:

Rebalancing your hdfs, hadoop cluster

Are you stuck in a scenario where replication factor is not correct, or not like what you expect it to be?

You can go for re balancing of hdfs, what you can do is:

Suppose you have replication factor as 2 but some files are showing replication factor as 2 or 3 or 1 and you want your replication factor to be 2.

Just increase the replication factor and then decrease for your hdfs root file system recursively.

suppose you need to increase the replication factor 2 and you are having replication factor as 1, and hadoop is not automatically replicating these blocks than you can increase the replication factor to 3 and then again decrease the replication factor to 2

Use following command to increase and then decrease the replication factor.

Increasing:

hadoop dfs -setrep -R 3 /      -----> this will increase the replication factors of all the files to 3 and replicate automatically once you have enough replication you can decrease the replication factor to stabalise the cluster as you needed

Decreaseing:

hadoop dfs -setrep -R 2 /     ------>  this will make the replication factor to 2 recursively for your hdfs root partition.

this method you can apply for a single file or a specific folder too.

if you have over and underutilized nodes in the hadoop cluster you can run balancer which is in bin dir to make your cluster balanced.

NOTE : you should have enough space on your dfs for replication, because as you are increasing the replication factor, it will need space.