Friday, July 26, 2013

The path "" is not a valid path to the 3.5.0-17-generic kernel headers / VMWare can't find linux headers path

Error/Problem while installing VMware Tool in Virtual Machine

If you this error falls in front of you (Ubuntu)

Throw this command infront of it :)

sudo apt-get install build-essential linux-headers-`uname -r` psmisc

-or-

sudo apt-get install linux-headers-$(uname -r)


If anyone of this command will executed successfully, then the error will run away from you :)

Thursday, July 11, 2013

Setting HeartBeat Interval for Datanode

Setting up the following in your hdfs-site.xml will give you 1-minute timeout.
<property>
 <name>heartbeat.recheck.interval</name>
 <value>15</value>
 <description>Determines datanode heartbeat interval in seconds</description>
</property>
If above doesn't work - try the following (seems to be version-dependent):
<property>
 <name>dfs.heartbeat.recheck.interval</name>
 <value>15</value>
 <description>Determines datanode heartbeat interval in seconds.</description>
</property>

Timeout equals to 2 * heartbeat.recheck.interval + 10 * heartbeat.interval. Default forheartbeat.interval is 3 seconds.

Monday, July 8, 2013

Thursday, July 4, 2013

How to copy files from one Hadoop Cluster to another ?

Suppose if you want to copy files from hadoop clusters you have three options :

1 : copy file to local and then copy from local using
  
 copyToLocal and then copyFromLocal
  
 -get and -put

But not a good option.

So we have another option:

-cp and distcp

Distcp will require Map-reduce to be running if you dont want to run Mapreduce on your cluster you have other option that is -cp

Uses:
hadoop dfs -cp hdfs://<source> hdfs://<destination>

if you want faster copy use distcp for that your job tracker and task tracker must be running.

distcp uses:

hadoop distcp hdfs://<source> hdfs://<destination>.

Featured Posts

#Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc

 #Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc Linux is an open-source operating system that is loved by millio...