Find and remove empty direcory

This command will find the empty directory and delete it recursively

find . -type d -empty -delete   --> Recursively

find . -empty -type d -exec rmdir {} +     --> From current Dir

find . -depth -type d -empty -exec rmdir -v {} +




Find all files of type <*.txt or what ever you like>

find . -name \*.txt -print    --> In place of *.txt you can specify extention                             
                                                       you like to search

java.net.SocketException Too many open files

Have you ever encountered this error? is so here is the solution.... :)

This may happen if there are many http request, or forgetting to close the open connection in time so for this what you can do is you can increase the number of open file in linux machine, as it is related to os issue mostly.

Open /etc/sysctl.conf and add the following :

fs.file-max = <65535>   <-- keep this number as you need

And for changing the effect of this use following command

sudo sysctl -p /etc/sysctl.conf

Find file of specific size Linux

Some time you many need to find all files greater than equal to specific size like 1MB 10 MB or 1GB, so you can use following command to get the files have length more than equal to you have specified:

find / -type f -size <GIVE SIZE HERE IN KB> -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

Example:

find / -type f -size +1048576k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'


This will find files greater than size 1GB. so you can experiment with the size as you need.

Whitelist a node in hadoop

You have a cluster with black listed nodes, this is how you can make dem white listed nodes

<property>
  <name>dfs.hosts</name>
  <value>path to whitelisted node file</value>
</property>

then issue following command

./bin/hadoop dfsadmin -refreshNodes

Hadoop Eco System

HDFS: The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets.

Search for a string in all files in a directory Linux

Grep command you can use to search a specified string in all files in the directory path you have given

grep "string to search" /var/*

so this will search string to search in /var/ directory files.

another variation :

find . -type f -exec grep -i "string to find" {} \; -print

or you can use

grep "string to search" *.htm    --> this will search for the string to search in all htm files


There can be more options as always :) try and find more.

Hadoop on Windows Azure

Setting Up Your Cluster On Windows :

Once you're invited to Participate in the beta, you can set up your Hadoop cluster. Go to hadooponazure.com and log in with your Windows Live ID Authorized. Next, fill out the dialog boxes on the Web site using the Following values:

1. Cluster (DNS) name: Enter name in the form "unique <your string>. CloudApp. Net".
2. Cluster size: Choose the number of nodes, from 4 to 32, and Their Associated storage allocations, from 2TB to 16TB per cluster.
3. Administrator username and password: Enter a username and password, password complexity restrictions are listed on the page. Once this is set you can connect via remote desktop or via Excel.
4. Configuration information for a SQL Azure instance: This is an option for storing the Hive MetaStore. If it's selected, you'll need to supply the URL to your SQL Azure server instance, as well as the name of the target database and login credentials. The login you specify must have the Following permissions on the target database: ddl_ddladmin, ddl_datawriter, ddl_datareader.

For more information click Link

Free Public DNS Server

OpenDNS free dns server list / IP address:

• 208.67.222.222
• 208.67.220.220

Google public dns server IP address:

• 8.8.8.8
• 8.8.4.4

Dnsadvantage free dns server list:

• 156.154.70.1
• 156.154.71.1

Linux : Date/Time Wise History, Date Wise command history in Linux

export HISTTIMEFORMAT="%m/%d - %H:%M:%S: --> "

It will change the HISTTIMEFORMAT variable and bash will store a timestamp in its history accordingly. Then your history will look like this

1  08/16 - 16:12:37: --> cat README | less
2 08/16 - 16:12:58: --> pkg-config --list-all | grep webkit
3 08/16 - 16:13:04: --> history

These are the date/time format you can use.

%d - Day
%m - Month
%y - Year
%T - Time
%H - Hours
%M - Minutes
%S - Seconds

org.apache.hadoop.hbase.regionserver.HRegionServer ABORTING region server Unhandled exception org.apache.hadoop.hbase.ClockOutOfSyncException

org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server : Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException:

This error occurs due to time synchronization between the nodes of the cluster, if the time difference between the node is more that 30000ms between the master and slave time then this error comes:

What is the solution ????

Setting up Standalone zookeeper cluster for production cluster

1.      Download the zookeeper from


a.      http://zookeeper.apache.org/releases.html#download or http://www.apache.org/dyn/closer.cgi/zookeeper/



2.      The downloaded file will be Tar file so extract in your desired directory.

3.      Create Directory for Zookeeper snapshot logs