Datanode Decommissioning from Hadoop Cluster

Hadoop offers the decommission feature to properly take out a set of existing data-nodes. The nodes to be taken out of cluster should be included into the exclude file, and the exclude file name should be specified as a configuration parameter dfs.hosts.exclude. This file should have been specified during namenode startup. It could be a zero length file. You must use the full hostname, ip or ip:port format in this file. Then the shell command

bin/hadoop dfsadmin -refreshNodes

Best Way to add nodes to hadoop cluster

Add the new node's DNS name to the conf/slaves file on the master node.  Then log in to the new slave node and execute:

If you are using Cloudera's distribution of Hadoop:

service hadoop-0.20-datanode start
service hadoop-0.20-tasktracker start

If you are using Apache distribution of Hadoop:

Configure MySQL as a MetaStore For Hive. MySQL as Hive

1. first your mysql should be installed and running.

I will show you how to configure mysql as a meta store for hive

lets start --->  

Ubuntu returning to login screen, Ubuntu Login loop

 

If you are having login problem in ubuntu, for example if you are putting your password and login screen coming again and again then just try following solution.

sudo apt-get install --reinstall xorg

/home/<username>/.XAuthority*    <—The user name which is not able to login.

then restart the system.

 

and if above does not work try following

switch to shell (Ctrl+Alt+F1) you can use <F1 to F6 in place of F1>

logged in as the user

cd /home/user 

sudo mv .Xauthority .XauthorityBak

sudo reboot

Install oracle/sun java on ubuntu 12.10

1. Open a terminal window.
2. Type in the following commands then hit Enter after each. 

• sudo sh -c "echo 'deb http://www.duinsoft.nl/pkg debs all' >> /etc/apt/sources.list"
• sudo apt-get update
• sudo apt-key adv --keyserver keys.gnupg.net --recv-keys 5CB26B26
• sudo apt-get update
• sudo apt-get install update-sun-jre

To install JDK 7 on i386 32-bit systems:

1. Open a terminal window.
2. Type in the following commands then hit Enter after each.
cd /tmp
wget -c --no-cookies --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com" "http://goo.gl/g9cJl" -O jdk-7u7-nb-7_2-linux-i586-ml.sh

• chmod +x jdk-7u7-nb-7_2-linux-i586-ml.sh
• sudo sh jdk-7u7-nb-7_2-linux-i586-ml.sh

To install JDK 7 on AMD 64-bit systems:

cd /tmp

wget -c --no-cookies --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com" "http://goo.gl/AJ1oS" -O jdk-7u7-nb-7_2-linux-x64-ml.sh

• chmod +x jdk-7u7-nb-7_2-linux-x64-ml.sh
• sudo sh jdk-7u7-nb-7_2-linux-x64-ml.sh

When the install is complete, use these commands:

• sudo mkdir -p /usr/lib/jvm/
• sudo cp -R /usr/local/jdk1.7.0* /usr/lib/jvm/
• sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.7.0_07/bin/javac 1
• sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.7.0_07/bin/java 1

For more, see the original article at the link below.
http://www.itworld.com/software/305913/install-oracle-java-7-ubuntu-1210
and
http://www.upubuntu.com/2012/10/how-to-install-oracle-java-7-jre-7-jdk.html?m=0

Open a port to listen on in Linux

The following command you can use to open a port to if you are getting a connection refused error on specific port number :

iptables -A INPUT -p tcp --dport <Port>2 -j ACCEPT

Hive Thrift Server

Hive provide a service that can be called as hive server or thrift server which helps us to access hive using different languages like c++, Java, Ruby, Python and many others remotely like how we use JDBC or ODBC connectors.

By default we use Command line to access hive, but to use hive programmatically we need such facility as a server that runs on an address and a port no so that it can be used in programming language efficiently and easily.

 

Starting Thrift Server :

 

bin/hive  - -service  hiveserver  &      -----> this will start thrift server and will give terminal to be used

or

bin/hive - - service hiveserver          -------> this need to keep the terminal open, if terminal is closed server will be killed (Look for & operator in linux)

 

 

after starting this you can verify whether hive thrift server is running or not, by using following command

netstat –nl | grep 10000

 

if it shows something like :

 

tcp 0 0 :::10000 :::* Listen

 

that means your thrift server is running successfully. by default thrift server runs on port no 10000 we and also make it to run on different port no. as follows

build/dist/bin/hive --service hiveserver --help
usage HIVE_PORT=xxxx ./hive --service hiveserver
  HIVE_PORT : Specify the server port

 

Limitation with Thrift server : HiveServer can not handle concurrent requests from more than one client. This is actually a limitation imposed by the Thrift interface that HiveServer exports, and can't be resolved by modifying the HiveServer code.

for using hive with jdbc and java refer –>  Infinity

ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried

ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7 times

Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

There may be more than one probable reason :  one is here :)

Check if namenode is in safemode....... If so wait for namenode to come out from safe mode or after waiting for 1 minute you can just ask name node to come out of safe mode using command

bin/hadoop dfsadmin -safemode leave

and then try to work with hbase ..

Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN.

The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.

The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system.

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

 

Check this LINK for more detail

Hadoop High Availability (HA hadoop Cluster)

Note: Currently, only manual failover is supported, means rely on the operator to manually initiate a failover. Automatic failure detection and initiation of a failover will be implemented in future versions.

The best technology has one demerit you know what : High availability of Namenode, so if NameNode is down whole cluster is down, so here hadoop has added HA feature in Hadoop which will make it more available for use i did not get any other word.

High Availability feature addresses the above problems by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This allows a fast failover to a new NameNode in the case that a machine crashes, or a graceful administrator-initiated failover for the purpose of planned maintenance.

How it is organised ?

Demystifying Hadoop concepts Series: Safe mode

What is is safe mode of hadoop, may time we come across this exception “ org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException”  or some other exceptions Which contains safe mode in it .

First let me tell what Safe mode is in context to Hadoop : as we all know Name node contains fsimage (metadata) of the data present on the cluster, which can be large or small based on the size of the cluster and the size of date present on the cluster, so when the name node starts it loads this fsimage and the edit logs from the disk in the Primary memory RAM for fast processing, and after loading it waits for data nodes to report about the present on those data nodes, so during this process that is loading the fsimage and edit logs and waiting for data nodes to report about the data block in safe mode, which is a read only mode for name node this is done to maintain the consistency of the data present, this is just like saying “ i will not receive any thing till i know what i already have”. And during this period no modification to the file blocks are allowed as to maintain the correctness of the data.