Host is in bad health Cloudera Manager after reinstalling. ERROR Error, CM server guid updated, expected

It happened to me when i tried to clean up and reinstall the Cloudera-manager tot error in web UI "Host is in bad health Cloudera Manager", i checked the log where is found following error:

ERROR    Error, CM server guid updated, expected e0c62792-a6a4-4070-b725-3ec1c7265e8e, received 03dfe52e-5beb-4794-8178-f5107739a3d6

This happened due to some leftover files which confuses cloudera agent and CM for this we need to do following

delete this file on all the nodes where ever you have the agent running(It will show in the error message that agent failed on following hosts) and restart the agent service

/var/lib/cloudera-scm-agent/cm_guid

and try there may be some other reason, so i request you to look into log of agent, server whenever you get error during installation you will surely get the pointer of the probable reason behind the error.

Jobtracker API error - Call to localhost/127.0.0.1:50030 failed on local exception: java.io.EOFException

Try the port number listed in your $HADOOP_HOME/conf/mapred-site.xml under the mapred.job.tracker property. Here's my pseudo mapred-site.xml conf

<property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
</property>

If you look at the JobTracker.getAddress(Configuration) method, you can see it uses this property if you don't explicitly specify the jobtracker host / port:

public static InetSocketAddress getAddress(Configuration conf) {
  String jobTrackerStr =
    conf.get("mapred.job.tracker", "localhost:8012");
  return NetUtils.createSocketAddr(jobTrackerStr);
}

Hadoop Eco System

HDFS: The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets.

Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN.

The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.

The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system.

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

 

Check this LINK for more detail

Demystifying Hadoop concepts Series: Safe mode

What is is safe mode of hadoop, may time we come across this exception “ org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException”  or some other exceptions Which contains safe mode in it .

First let me tell what Safe mode is in context to Hadoop : as we all know Name node contains fsimage (metadata) of the data present on the cluster, which can be large or small based on the size of the cluster and the size of date present on the cluster, so when the name node starts it loads this fsimage and the edit logs from the disk in the Primary memory RAM for fast processing, and after loading it waits for data nodes to report about the present on those data nodes, so during this process that is loading the fsimage and edit logs and waiting for data nodes to report about the data block in safe mode, which is a read only mode for name node this is done to maintain the consistency of the data present, this is just like saying “ i will not receive any thing till i know what i already have”. And during this period no modification to the file blocks are allowed as to maintain the correctness of the data.

HBase Security for the Enterprise

Trend Micro developed the new security features in HBase 0.92 and has the first known deployment of secure HBase in production. We will share our motivations, use cases, experiences, and provide a 10 minute tutorial on how to set up a test secure HBase cluster and a walk through of a simple usage example. The tutorial will be carried out live on an on-demand EC2 cluster, with a video backup in case of network or EC2 unavailability.

Source : here

What is the difference between HDFS and NAS ?

The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. Following are differences between HDFS and NAS
• In HDFS Data Blocks are distributed across local drives of all machines in a cluster. Whereas in NAS data is stored on dedicated hardware.
• HDFS is designed to work with Map Reduce System, since computation are moved to data. NAS is not suitable for Map Reduce since data is stored separately from the computations.
• HDFS runs on a cluster of machines and provides redundancy using replication protocol. Whereas NAS is provided by a single machine therefore does not provide data redundancy.

What is a Job Tracker in Hadoop? How many instances of Job Tracker run on a Hadoop Cluster?

Job Tracker is the daemon service for submitting and tracking Map Reduce jobs in Hadoop. There is only One Job Tracker process run on any hadoop cluster. Job Tracker runs on its own JVM process. In a typical production cluster its run on a separate machine. Each slave node is configured with job tracker node location. The Job Tracker is single point of failure for the Hadoop Map Reduce service. If it goes down, all running jobs are halted. Job Tracker in Hadoop performs following actions(from Hadoop Wiki:)
• Client applications submit jobs to the Job tracker.
• The JobTracker talks to the NameNode to determine the location of the data
• The JobTracker locates TaskTracker nodes with available slots at or near the data
• The JobTracker submits the work to the chosen TaskTracker nodes.
• The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker.
• A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable.
• When the work is completed, the JobTracker updates its status.
• Client applications can poll the JobTracker for information.

Name node or Data node not starting

If you come across these problems :

Data node not starting :

          Solution you can try :

              1.  "chown" of hdfs folderr to the current user

              2.  " chmod" of hdfs folder to 755 to the current user

              3.  delete the hdfs folder and reformat (hadoop namenode -format) if 1 and 2 does not work

     Note : Problem with data node generally occurs due to file permission.

Name node not starting

          Solution you can try:

               1. Comment out or remove the entry 127.0.1.1

               2. Check your ip/dns setting, name node problem generally related to domain/IP

              3. Also check if you name/IP you are specifying in the hadoop setting xml can be resolved. you can do this bye pinging/ or telnetting the IP/Domain you have specified for the hadoop settings.

     Note : Problem related to name node generally occurs due to domain/IP related problems.

     

Why to choose HBase and What HDFS Gives Us

 

Hadoop Shell Commands

namenode -format	format the DFS filesystem
secondarynamenode	run the DFS secondary namenode
namenode	run the DFS namenode
datanode	run a DFS datanode
dfsadminmradmin	run a DFS admin client
mradmin	run a Map-Reduce admin client

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException

hadoop dfsadmin -safemode leave

If you get "org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory" exception run the above command for this is just a temperary solution. For permanent solution just check the permission of your hadoop folder permissions. make it to 755

Copy file from local disk to hdfs using java

Local File ====== File Copy =======> hdfs file system

Create a project in eclipse or netbeans or any editor you like and add hadoop-core.jar and create a class named PutToHdfs and put this code

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * 
 * @author shashwat
 */
public class PutToHdfs {

Install Hadoop

Install Hadoop on a Single Node

1. Install an instance of Amazon EC2 with Amazon Linux
   
   A micro EC2 instance may not have enough memory to run Hadoop in pseudo-distributed mode
2. Login to the created instance
3. Install & upgrade software needed by Hadoop
   

   cd
   sudo yum install rsync
   sudo yum install java-1.6.0-openjdk

4. Download and install the latest stable Hadoop
   

   wget sudo http://mirror.its.uidaho.edu/pub/apache//hadoop/core/stable/hadoop-0.20....
   tar -zxvf hadoop-0.20.2.tar.gz

   Replace the URL with the latest stable version
   
5. Move the installation to home
   

   sudo mv hadoop-0.20.2 /home

6. (Optional) For better security, create a new user to own the Hadoop
   

   sudo useradd -s /bin/bash -m hdp
   sudo chown -R hdp.hdp /home/hadoop-0.20.2

7. Test hadoop installation
   

   sudo su hdp
   cd /home/hadoop-0.20.2
   bin/hadoop

Hadoop Standalone Operation for Debugging

HDFS Permissions

Overview

The Hadoop Distributed File System (HDFS) implements a permissions model for files and directories that shares much of the POSIX model. Each file and directory is associated with an owner and a group. The file or directory has separate permissions for the user that is the owner, for other users that are members of the group, and for all other users. For files, the r permission is required to read the file, and the wpermission is required to write or append to the file. For directories, the r permission is required to list the contents of the directory, the w permission is required to create or delete files or directories, and the xpermission is required to access a child of the directory. In contrast to the POSIX model, there are no sticky, setuid or setgid bits for files as there is no notion of executable files. For directories, there no sticky,setuid or setgid bits directory as a simplification. Collectively, the permissions of a file or directory are its mode. In general, Unix customs for representing and displaying modes will be used, including the use of octal numbers in this description. When a file or directory is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule).

Each client process that accesses HDFS has a two-part identity composed of the user name, and groups list. Whenever HDFS must do a permissions check for a file or directory foo accessed by a client process,

• If the user name matches the owner of foo, then the owner permissions are tested;
• Else if the group of foo matches any of member of the groups list, then the group permissions are tested;
• Otherwise the other permissions of foo are tested.

If a permissions check fails, the client operation fails.

User Identity

In this release of Hadoop the identity of a client process is just whatever the host operating system says it is. For Unix-like systems,

• The user name is the equivalent of `whoami`;
• The group list is the equivalent of `bash -c groups`.

In the future there will be other ways of establishing user identity (think Kerberos, LDAP, and others). There is no expectation that this first method is secure in protecting one user from impersonating another. This user identity mechanism combined with the permissions model allows a cooperative community to share file system resources in an organized fashion.

In any case, the user identity mechanism is extrinsic to HDFS itself. There is no provision within HDFS for creating user identities, establishing groups, or processing user credentials.

HBase MapReduce Read/Write Example

Configuration config = HBaseConfiguration.create();
Job job = new Job(config,"ExampleReadWrite");
job.setJarByClass(MyReadWriteJob.class);    // class that contains mapper
                       
Scan scan = new Scan();
scan.setCaching(500);        // 1 is the default in Scan, which will be bad for MapReduce jobs
scan.setCacheBlocks(false);  // don't set to true for MR jobs
// set other scan attrs
           

HBase MapReduce Read Example

Configuration config = HBaseConfiguration.create();
Job job = new Job(config, "ExampleRead");
job.setJarByClass(MyReadJob.class);     // class that contains mapper
   
Scan scan = new Scan();
scan.setCaching(500);        // 1 is the default in Scan, which will be bad for MapReduce jobs
scan.setCacheBlocks(false);  // don't set to true for MR jobs
// set other scan attrs

Running Hadoop in Pseudo Distributed Mode

This section contains instructions for Hadoop installation on ubuntu. This is Hadoop quickstart tutorial to setup Hadoop quickly. This is shortest tutorial of Hadoop installation, here you will get all the commands and their description required to install Hadoop in Pseudo distributed mode(single node cluster)

COMMAND	DESCRIPTION
sudo apt-get install sun-java6-jdk	Install java
	If you don't have hadoop bundle download here download hadoop
sudo tar xzf file_name.tar.gz	Extract hadoop bundle
Go to your hadoop installation directory(HADOOP_HOME)
vi conf/hadoop-env.sh	Edit configuration file hadoop-env.sh and set JAVA_HOME: export JAVA_HOME=path to be the root of your Java installation(eg: /usr/lib/jvm/java-6-sun)
vi conf/core-site.xml then type:  <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration>	Edit configuration file core-site.xml
vi conf/hdfs-site.xml then type:  <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>	Edit configuration file hdfs-site.xml
vi conf/mapred.xml then type: <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> </configuration>	Edit configuration file mapred-site.xml and type:
sudo apt-get install openssh-server openssh-client	install ssh
ssh-keygen -t rsa -P "" cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys ssh localhost	Setting passwordless ssh
bin/hadoop namenode –format	Format the new distributed-filesystem During this operation : Name node get start Name node get formatted Name node get stopped
bin/start-all.sh	Start the hadoop daemons
jps	It should give output like this: 14799 NameNode 14977 SecondaryNameNode 15183 DataNode 15596 JobTracker 15897 TaskTracker
Congratulations Hadoop Setup is Completed
http://localhost:50070/	web based interface for name node
http://localhost:50030/	web based interface for job tracker
Now lets run some examples
bin/hadoop jar hadoop-*-examples.jar pi 10 100	run pi example
bin/hadoop dfs -mkdir input bin/hadoop dfs -put conf input bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' bin/hadoop dfs -cat output/*	run grep example
bin/hadoop dfs -mkdir inputwords bin/hadoop dfs -put conf inputwords bin/hadoop jar hadoop-*-examples.jar wordcount inputwords outputwords bin/hadoop dfs -cat outputwords/*	run wordcount example
bin/stop-all.sh	Stop the hadoop daemons

Quick install HBase in “pseudo distributed” mode and connect from Java

On first reading of the HBase documentation, setting up pseudo distributed mode sounds very simple. The problem is that there a lot of gotchas, which can make life very difficult indeed. Consequently, many people follow a twisted journey to their final destination, and when they finally get there, aren’t sure which of the measures they took were needed, and which were not. This is reflected by a degree of misinformation on the Web, and I will try and present here a reasonably minimal way of getting up and running (that is not even to say that every step I take is absolutely necessary even, but I’ll mention where I’m not sure).

Step 1: Check your IP setup
I believe this is one of the main causes of the weirdness that can happen. So, if you’re on Ubuntu check your hosts file. If you see something like:
127.0.0.1 localhost
127.0.1.1 <server fqn> <server name, as in /etc/hostname>
get rid of the second line, and change to
127.0.0.1 locahost
<server ip> <server fqn> <server name, as in /etc/hostname>
e.g.
127.0.0.1 localhost
23.201.99.100 hbase.mycompany.com hbase

Hadoop Troubleshooting

General Advice

• If you are having problems, check the logs in the logs directory to see if there are any Hadoop errors or Java Exceptions.
• Logs are named by machine and job they carry out in the cluster, and this can help you figure out which part of your configuration is giving you trouble.
• Even if you were very careful, the problem is probably with your configuration. Try running the grep example from the QuickStart. If it doesn't run then you need to check your configuration.
• If you can't get it to work on a real cluster, try it on a single-node.
• Sometimes it can just take some time and sweat to make complex systems run; but, it never hurts to ask for help so please ask the TA and your fellow students ASAP if you are having trouble making Hadoop run.

Symptoms and Possible Solutions

Symptom	Possible Problem	Possible Solution
You get an error that you cluster is in "safe mode"	Your cluster enters safe mode when it hasn't been able to verify that all the data nodes necessary to replicate your data are up and responding. Checkthe documentation to learn more about safe mode.	First, wait a minute or two and then retry your command. If you just started your cluster, it's possible that it isn't fully initialized yet. If waiting a few minutes didn't help and you still get a "safe mode" error, check your logs to see if any of your data nodes didn't start correctly (either they have Java exceptions in their logs or they have messages stating that they are unable to contact some other node in your cluster). If this is the case you need to resolve the configuration issue (or possibly pick some new nodes) before you can continue.
You get a NoRouteToHostException in your logs or in stderr output from a command.	One of your nodes cannot be reached correctly. This may be a firewall issue, so you should report it to me.	The only workaround is to pick a new node to replace the unreachable one. Currently, I think that creusa is unreachable, but all other Linux boxes should be okay. None of the Macs will currently work in a cluster.
You get an error that "remote host identification has changed" when you try to ssh to localhost.	You have moved your single node cluster from one machine in the Berry Patch to another. The name localhost thus is pointing to a new machine, and your ssh client thinks that it might be a man-in-the-middle attack.	You can ask your login to skip checking the validity of localhost. You do this by setting NoHostAuthenticationForLocalhost to yes in ~/.ssh/config. You can accomplish this with the following command: echo "NoHostAuthenticationForLocalhost yes" >>~/.ssh/config
Your DataNode is started and you can create directories withbin/hadoop dfs -mkdir, but you get an error message when you try to put files into the HDFS (e.g., when you run a command like bin/hadoop dfs -put).	Creating directories is only a function of the NameNode, so your DataNode is not exercised until you actually want to put some bytes into a file. If you are sure that the DataNode is started, then it could be that your DataNodes are out of disk space.	Go to the HDFS info web page (open your web browser and go to http://namenode:dfs_info_port where namenode is the hostname of your NameNode and dfs_info_port is the port you chose dfs.info.port; if followed the QuickStart on your personal computer then this URL will be http://localhost:50070). Once at that page click on the number where it tells you how many DataNodes you have to look at a list of the DataNodes in your cluster. If it says you have used 100% of your space, then you need to free up room on local disk(s) of the DataNode(s). If you are on Windows then this number will not be accurate (there is some kind of bug either in Cygwin's df.exe or in Windows). Just free up some more space and you should be okay. On one Windows machine we tried the disk had 1GB free but Hadoop reported that it was 100% full. Then we freed up another 1GB and then it said that the disk was 99.15% full and started writing data into the HDFS again. We encountered this bug on Windows XP SP2.
You try to run the grep example from the QuickStart but you get an error message like this: java.io.IOException: Not a file: hdfs://localhost:9000/user/ross/input/conf	You may have created a directory inside theinput directory in the HDFS. For example, this might happen if you run bin/hadoop dfs -put conf input twice in a row (this would create a subdirectory in input... why?).	The easiest way to get the example run is to just start over and make the input anew. bin/hadoop dfs -rmr input bin/hadoop dfs -put conf input
Your DataNodes won't start, and you see something like this inlogs/*datanode*: Incompatible namespaceIDs in /tmp/hadoop-ross/dfs/data	Your Hadoop namespaceID became corrupted. Unfortunately the easiest thing to do reformat the HDFS.	You need to do something like this: bin/stop-all.sh rm -Rf /tmp/hadoop-your-username/* bin/hadoop namenode -format Be VERY careful with rm -Rf
When you try the grep example in the QuickStart, you get an error like the following: org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : /user/ross/input	You haven't created an input directory containing one or more text files.	bin/hadoop dfs -put conf input
When you try the grep example in the QuickStart, you get an error like the following: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /user/ross/output already exists	You might have already run the example once, creating an output directory. Hadoop doesn't like to overwrite files.	Remove the output directory before rerunning the example: bin/hadoop dfs -rmr output Alternatively you can change the output directory of the grep example, something like this: bin/hadoop jar hadoop-*-examples.jar \ grep input output2 'dfs[a-z.]+'
You can run Hadoop jobs written in Java (like the grep example), but your HadoopStreaming jobs (such as the Python example that fetches web page titles) won't work.	You might have given only a relative path to the mapper and reducer programs. The tutorial originally just specified relative paths, but absolute paths are required if you are running in a real cluster.	Use absolute paths like this from the tutorial: bin/hadoop jar contrib/hadoop-0.15.2-streaming.jar \ -mapper $HOME/proj/hadoop/multifetch.py \ -reducer $HOME/proj/hadoop/reducer.py \ -input urls/* \ -output titles