Tuesday, June 16, 2015

Extract All Tar Files in a directory in Linux

This will first list the files containing extension tar.gz, and then awk will get the file names which is column 9 in

ls -lrth command, and

NF > 2 will remove the blank line and

tar -xvzf will extract files names contained in variable $i,

Like this we can experiment various operations like renaming all files with specific extension of so and can fiddled with to achieve various goals.


for i in `ls -lrth *.tar.gz |awk  'NF>2 {print $9}'`; do tar -xvzf $i; done

Thursday, June 4, 2015

All about hadoop Balancer.


Hadoop Data Balancing
Hadoop Data Balancing



Hadoop Balancer:

This is tool provided to balance the disk uses throughout the Hadoop cluster. I may happen sometime that some of the nodes in the cluster becomes over utilized or underutilized, which occurs due to addition of new nodes where newly added nodes may be underutilized or if there are less number of nodes result in overutilization. We can run balancer from more than 1 machine in the cluster to increase the speed of balancing but it will increase bandwidth uses to very high.
This tool requires administrator right on the Hadoop cluster to run.




Syntax of the balancer:

bin/start-balancer.sh [-threshold <threshold>]

Where start-balancer.sh files resides in the bin directory of the Hadoop folder. And the threshold is the parameter which decides target of balance, this lies in fraction between 0,1 the default value is 10% if nothing is passed as the threshold value.

This process does the transferring of blocks between the nodes resulting network activity and if a production cluster must be used cautiously, as it result in some block missing error or late reply from the cluster.
This process can be stopped any time if required using following command:

Featured Posts

#Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc

 #Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc Linux is an open-source operating system that is loved by millio...