Hadoop fully distributed cluster configuration 1,2,3

Install Hadoop

Install Hadoop on a Single Node

1. Install an instance of Amazon EC2 with Amazon Linux
   
   A micro EC2 instance may not have enough memory to run Hadoop in pseudo-distributed mode
2. Login to the created instance
3. Install & upgrade software needed by Hadoop
   

   cd
   sudo yum install rsync
   sudo yum install java-1.6.0-openjdk

4. Download and install the latest stable Hadoop
   

   wget sudo http://mirror.its.uidaho.edu/pub/apache//hadoop/core/stable/hadoop-0.20....
   tar -zxvf hadoop-0.20.2.tar.gz

   Replace the URL with the latest stable version
   
5. Move the installation to home
   

   sudo mv hadoop-0.20.2 /home

6. (Optional) For better security, create a new user to own the Hadoop
   

   sudo useradd -s /bin/bash -m hdp
   sudo chown -R hdp.hdp /home/hadoop-0.20.2

7. Test hadoop installation
   

   sudo su hdp
   cd /home/hadoop-0.20.2
   bin/hadoop

Hadoop Standalone Operation for Debugging

HDFS Permissions

Overview

The Hadoop Distributed File System (HDFS) implements a permissions model for files and directories that shares much of the POSIX model. Each file and directory is associated with an owner and a group. The file or directory has separate permissions for the user that is the owner, for other users that are members of the group, and for all other users. For files, the r permission is required to read the file, and the wpermission is required to write or append to the file. For directories, the r permission is required to list the contents of the directory, the w permission is required to create or delete files or directories, and the xpermission is required to access a child of the directory. In contrast to the POSIX model, there are no sticky, setuid or setgid bits for files as there is no notion of executable files. For directories, there no sticky,setuid or setgid bits directory as a simplification. Collectively, the permissions of a file or directory are its mode. In general, Unix customs for representing and displaying modes will be used, including the use of octal numbers in this description. When a file or directory is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule).

Each client process that accesses HDFS has a two-part identity composed of the user name, and groups list. Whenever HDFS must do a permissions check for a file or directory foo accessed by a client process,

• If the user name matches the owner of foo, then the owner permissions are tested;
• Else if the group of foo matches any of member of the groups list, then the group permissions are tested;
• Otherwise the other permissions of foo are tested.

If a permissions check fails, the client operation fails.

User Identity

In this release of Hadoop the identity of a client process is just whatever the host operating system says it is. For Unix-like systems,

• The user name is the equivalent of `whoami`;
• The group list is the equivalent of `bash -c groups`.

In the future there will be other ways of establishing user identity (think Kerberos, LDAP, and others). There is no expectation that this first method is secure in protecting one user from impersonating another. This user identity mechanism combined with the permissions model allows a cooperative community to share file system resources in an organized fashion.

In any case, the user identity mechanism is extrinsic to HDFS itself. There is no provision within HDFS for creating user identities, establishing groups, or processing user credentials.

Running Hadoop in Pseudo Distributed Mode

This section contains instructions for Hadoop installation on ubuntu. This is Hadoop quickstart tutorial to setup Hadoop quickly. This is shortest tutorial of Hadoop installation, here you will get all the commands and their description required to install Hadoop in Pseudo distributed mode(single node cluster)

COMMAND	DESCRIPTION
sudo apt-get install sun-java6-jdk	Install java
	If you don't have hadoop bundle download here download hadoop
sudo tar xzf file_name.tar.gz	Extract hadoop bundle
Go to your hadoop installation directory(HADOOP_HOME)
vi conf/hadoop-env.sh	Edit configuration file hadoop-env.sh and set JAVA_HOME: export JAVA_HOME=path to be the root of your Java installation(eg: /usr/lib/jvm/java-6-sun)
vi conf/core-site.xml then type:  <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration>	Edit configuration file core-site.xml
vi conf/hdfs-site.xml then type:  <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>	Edit configuration file hdfs-site.xml
vi conf/mapred.xml then type: <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> </configuration>	Edit configuration file mapred-site.xml and type:
sudo apt-get install openssh-server openssh-client	install ssh
ssh-keygen -t rsa -P "" cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys ssh localhost	Setting passwordless ssh
bin/hadoop namenode –format	Format the new distributed-filesystem During this operation : Name node get start Name node get formatted Name node get stopped
bin/start-all.sh	Start the hadoop daemons
jps	It should give output like this: 14799 NameNode 14977 SecondaryNameNode 15183 DataNode 15596 JobTracker 15897 TaskTracker
Congratulations Hadoop Setup is Completed
http://localhost:50070/	web based interface for name node
http://localhost:50030/	web based interface for job tracker
Now lets run some examples
bin/hadoop jar hadoop-*-examples.jar pi 10 100	run pi example
bin/hadoop dfs -mkdir input bin/hadoop dfs -put conf input bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' bin/hadoop dfs -cat output/*	run grep example
bin/hadoop dfs -mkdir inputwords bin/hadoop dfs -put conf inputwords bin/hadoop jar hadoop-*-examples.jar wordcount inputwords outputwords bin/hadoop dfs -cat outputwords/*	run wordcount example
bin/stop-all.sh	Stop the hadoop daemons

Cluster Setup HADOOP

This document describes how to install, configure and manage non-trivial Hadoop clusters ranging from a few nodes to extremely large clusters with thousands of nodes.

To play with Hadoop, you may first want to install Hadoop on a single machine (see Hadoop Quick Start).

Pre-requisites

1. Make sure all requisite software is installed on all nodes in your cluster.
2. Get the Hadoop software.

Hadoop : Cluster Setup

Prerequisites

1. Make sure all required software is installed on all nodes in your cluster.
2. Download the Hadoop software.

Installation

Installing a Hadoop cluster typically involves unpacking the software on all the machines in the cluster.

Typically one machine in the cluster is designated as the NameNode and another machine the as JobTracker, exclusively. These are the masters. The rest of the machines in the cluster act as both DataNode and TaskTracker. These are the slaves.

The root of the distribution is referred to as HADOOP_HOME. All machines in the cluster usually have the same HADOOP_HOME path.