ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried

ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7 times

Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

There may be more than one probable reason :  one is here :)

Check if namenode is in safemode....... If so wait for namenode to come out from safe mode or after waiting for 1 minute you can just ask name node to come out of safe mode using command

bin/hadoop dfsadmin -safemode leave

and then try to work with hbase ..

HBase components and Know what......

It is one of the cool projects from Apache, that enable a facility to provide a large scale scale able, distributed database, based on Hadoop. In this data is organised as row columns that can grow infinitely as you add up new nodes. No need to reconfigure and mess up much with the configuration setting.

This requires Java and Hadoop to run full fledged manner.

Components:

HBaseMaster :

The HBaseMaster is responsible for assigning regions to HRegionServers. The first region to be assigned is the ROOT region which locates all the META regions to be assigned. The HBaseMaster also monitors the health of each HRegionServer, and if it detects a HRegionServer is no longer reachable, it will split the HRegionServer's write-ahead log so that there is now one write-ahead log for each region that the HRegionServer was serving. After it has accomplished this, it will reassign the regions that were being served by the unreachable HRegionServer. In addition, the HBaseMaster is also responsible for handling table administrative functions such as on/off-lining of tables, changes to the table schema (adding and removing column families), etc.

HRegionServer:

The HRegionServer is responsible for handling client read and write requests. It communicates with the BaseMaster to get a list of regions to serve and to tell the master that it is alive. Region assignments and

other instructions from the master "piggy back" on the heart beat messages.

HBase client:

The HBase client is responsible for finding HRegionServers that are serving the particular row range of interest. On instantiation, the HBase client communicates with the HBaseMaster to find the location of the ROOT region. This is the only communication between the client and the master.

Inherited from  : Here 

From : Research paper of (Ankur Khetrapal, Vinay Ganesh)

Secure HBase : Access Controls

Building and maintaining an HBase cluster is a complex undertaking, especially if you build your own hardware infrastructure. Leasing from a cloud service such as Amazon EC2 allows you to avoid the expense and complexity of setting up your own hardware, but you’ll still need to know how to install, configure and tune your HBase cluster on top of your leased instances.

But what if you could simply connect to a HBase instance, hosted in a public cloud, and let someone else worry about HBase setup and maintenance? We believe there’s a group of potential HBase users who simply want to connect to a managed HBase cluster and start storing their data.

Another class of customers may be large organizations that want to centralize IT resources within a private cloud: a single company-internal cluster running HBase. Such organizations may have several departments, each of which is a tenant in the private cloud.

Both of these groups of potential HBase users want to keep their data secure in the presence of other tenants of the hosted HBase cluster: they want own their own tables, and provide defined access to users in their department, and perhaps even provide defined access to other tenants.

 

Read Full Story

HBase Security for the Enterprise

Trend Micro developed the new security features in HBase 0.92 and has the first known deployment of secure HBase in production. We will share our motivations, use cases, experiences, and provide a 10 minute tutorial on how to set up a test secure HBase cluster and a walk through of a simple usage example. The tutorial will be carried out live on an on-demand EC2 cluster, with a video backup in case of network or EC2 unavailability.

Source : here

What is the difference between HDFS and NAS ?

The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. Following are differences between HDFS and NAS
• In HDFS Data Blocks are distributed across local drives of all machines in a cluster. Whereas in NAS data is stored on dedicated hardware.
• HDFS is designed to work with Map Reduce System, since computation are moved to data. NAS is not suitable for Map Reduce since data is stored separately from the computations.
• HDFS runs on a cluster of machines and provides redundancy using replication protocol. Whereas NAS is provided by a single machine therefore does not provide data redundancy.

What is a Job Tracker in Hadoop? How many instances of Job Tracker run on a Hadoop Cluster?

Job Tracker is the daemon service for submitting and tracking Map Reduce jobs in Hadoop. There is only One Job Tracker process run on any hadoop cluster. Job Tracker runs on its own JVM process. In a typical production cluster its run on a separate machine. Each slave node is configured with job tracker node location. The Job Tracker is single point of failure for the Hadoop Map Reduce service. If it goes down, all running jobs are halted. Job Tracker in Hadoop performs following actions(from Hadoop Wiki:)
• Client applications submit jobs to the Job tracker.
• The JobTracker talks to the NameNode to determine the location of the data
• The JobTracker locates TaskTracker nodes with available slots at or near the data
• The JobTracker submits the work to the chosen TaskTracker nodes.
• The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker.
• A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable.
• When the work is completed, the JobTracker updates its status.
• Client applications can poll the JobTracker for information.

Why to choose HBase and What HDFS Gives Us

 

Commands Available on HBase Shell

Groups of commands and explanation :

1. General Commands:
   1. status : Shows server status, example
      1 servers, 0 dead, 10.0000 average load
   2. version : Shows version of hbase
      0.90.4, r1150278, Sun Jul 24 15:53:29 PDT 2011
2. DDL Commands:
   1. alter : Using this commmand you can alter the table

Alter column family schema; pass table name and a dictionary

specifying new column family schema. Dictionaries are described

on the main help command output. Dictionary must include name

of column family to alter. For example,

To change or add the 'f1' column family in table 't1' from defaults

to instead keep a maximum of 5 cell VERSIONS, do:

hbase> alter 't1', NAME => 'f1', VERSIONS => 5

To delete the 'f1' column family in table 't1', do:

hbase> alter 't1', NAME => 'f1', METHOD => 'delete'

or a shorter version:

hbase> alter 't1', 'delete' => 'f1'

You can also change table-scope attributes like MAX_FILESIZE

MEMSTORE_FLUSHSIZE, READONLY, and DEFERRED_LOG_FLUSH.

For example, to change the max size of a family to 128MB, do:

hbase> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '134217728'

There could be more than one alteration in one command:

hbase> alter 't1', {NAME => 'f1'}, {NAME => 'f2', METHOD => 'delete'}

Impliment And/Or multi condition in hbase.

Here  FilterList listOfFiltersA and FilterList listOfFiltersB will impliment AND condition of the filters in the filter list and  FilterList listOfFiltersC will impliment OR Condition on  listOfFiltersA and listOfFiltersB

 public static void multiConditionReturn() throws IOException
    {
        HTable hTable = new HTable("well");
        Scan scan = new Scan();
        FilterList listOfFiltersA = new FilterList(FilterList.Operator.MUST_PASS_ALL);
        FilterList listOfFiltersB = new FilterList(FilterList.Operator.MUST_PASS_ALL);
        FilterList listOfFiltersC = new FilterList(FilterList.Operator.MUST_PASS_ONE);
        SingleColumnValueFilter nodeFilter;
       
        nodeFilter = new SingleColumnValueFilter
                (
                Bytes.toBytes("cf"),
                Bytes.toBytes("well_name"),
                CompareOp.EQUAL,
                Bytes.toBytes("AB Test Well")
                );
        listOfFiltersA.addFilter(nodeFilter);
        nodeFilter = new SingleColumnValueFilter
                (
                Bytes.toBytes("cf"),
                Bytes.toBytes("well_region"),
                CompareOp.EQUAL,
                Bytes.toBytes("Region Name")
                );
        listOfFiltersA.addFilter(nodeFilter);
       
        nodeFilter = new SingleColumnValueFilter
                (
                Bytes.toBytes("cf"),
                Bytes.toBytes("wellDatum_name"),
                CompareOp.EQUAL,
                Bytes.toBytes("Kelly Bushing")
                );
        listOfFiltersB.addFilter(nodeFilter);       
        nodeFilter = new SingleColumnValueFilter
                (
                Bytes.toBytes("cf"),
                Bytes.toBytes("well_uid"),
                CompareOp.EQUAL,
                Bytes.toBytes("BGC_TD")
                );
        listOfFiltersB.addFilter(nodeFilter);      
        listOfFiltersC.addFilter(listOfFiltersA);
        listOfFiltersC.addFilter(listOfFiltersB);
        scan.setFilter(listOfFiltersC);
        ResultScanner scanner = hTable.getScanner(scan);

        for (Result result : scanner) {
        System.out.println(result);           
        }
    }

How to delete a specific row or given rows in hbase using java code

/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */
package witsml131;

import java.io.IOException;
import java.util.ArrayList;
import javax.security.auth.login.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.util.Bytes;

/**
 *
 * @author shashwat
 */
public class DeleteFromHbase {

    public static void main(String[] args) throws IOException {

        HTable table = new HTable("well");
        ArrayList<String> rowsToDelete = new ArrayList(); //array with all rows names to delete.
        Delete del = new Delete(Bytes.toBytes("2"));//for single row named 2

        table.delete(del);
      
        //deleting all rows in rowsToDelete in array list rowsToDelete
        for (String rows : rowsToDelete) {
            del = new Delete(Bytes.toBytes(rows));//for single row named 2

            table.delete(del);
        }
        table.close();
    }
}

Hive integration: HBase and Rcfile HadoopSummit2010

HBase Schema Design - Things you need to know

 

When designing schemas for HBase, be it from scratch or porting an existing application over from a relational database for example, there are a set of architectural constraints that a user should be aware of to avoid common pitfalls. This session discusses the basic underlying concepts of the storage

Working with NoSQL Databases

How to get started with NoSQL?

Since 2009 NoSQL databases becomes more and more popular. But why?

• No usage of SQL, that means
  • Less complexity
  • Better portability
  • Boundlessness
  • User-friendliness
• Most databases are Open-Source
• Performance
• Scalability

Famous companies like Twitter, Facebook and Amazon are using NoSQL databases.
What sorts of NoSQL databases are used today?

• Key-Value stores
  • Easy to implement
  • Only key-value-pairs can be stored
  • Difficult to build complex data structures
• Column stores
  • Columns don't have to be defined in advance.
  • A row can have different numbers of cells
• Document stores
  • Like key-value stores, but allows nested values
• Graph databases
  • Objects and relationships are modelled and persisted as nodes and edges of a graph

What NoSQL databases are present today?

Quick install HBase in “pseudo distributed” mode and connect from Java

On first reading of the HBase documentation, setting up pseudo distributed mode sounds very simple. The problem is that there a lot of gotchas, which can make life very difficult indeed. Consequently, many people follow a twisted journey to their final destination, and when they finally get there, aren’t sure which of the measures they took were needed, and which were not. This is reflected by a degree of misinformation on the Web, and I will try and present here a reasonably minimal way of getting up and running (that is not even to say that every step I take is absolutely necessary even, but I’ll mention where I’m not sure).

Step 1: Check your IP setup
I believe this is one of the main causes of the weirdness that can happen. So, if you’re on Ubuntu check your hosts file. If you see something like:
127.0.0.1 localhost
127.0.1.1 <server fqn> <server name, as in /etc/hostname>
get rid of the second line, and change to
127.0.0.1 locahost
<server ip> <server fqn> <server name, as in /etc/hostname>
e.g.
127.0.0.1 localhost
23.201.99.100 hbase.mycompany.com hbase

Easy Install HBase/Hadoop in Pseudo Distributed Mode

Introduction

This documentation should get you up and running with a full pseudo distributed Hadoop/HBase installation in an Ubuntu VM quickly. I use Ubuntu because the Debian Package Management (apt) is by far the best way to install software on a machine. It is possible to also use this on regular hardware as well.

The reason why you will need this is because much of the existing documentation is spread around quite a few different locations. Thus, I've already done the work of digging this information out so that you don't have to.

This documentation is intended to be read and used from top to bottom. Before you do an initial install, I suggest you read through it once first.

Reference Manuals

• https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation
• https://ccp.cloudera.com/display/CDHDOC/CDH3+Deployment+in+Pseudo-Distributed+Mode
• https://ccp.cloudera.com/display/CDHDOC/ZooKeeper+Installation
• https://ccp.cloudera.com/display/CDHDOC/HBase+Installation
• http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#PseudoDistributed
• http://hbase.apache.org/book.html
• http://hbase.apache.org/pseudo-distributed.html

Create the virtual machine

The first thing that you will want to do is download a copy of the Ubuntu Server 10.04 64bit ISO image. This version is the current Long Term Support (LTS) version. These instructions may work with a newer version, but I'm suggesting the LTS because that is what I test with and also what your operations team will most likely want to install into production. Once you have the ISO, create a new virtual machine using your favorite VM manager (I like vmware fusion on my Mac).

Hadoop : HBase

Copyright © 2011 Apache Software Foundation

Chapter 1. Getting Started

Table of Contents

1.1. Introduction

1.2. Quick Start

1.2.1. Download and unpack the latest stable release.

1.2.2. Start HBase

1.2.3. Shell Exercises

1.2.4. Stopping HBase

1.2.5. Where to go next

1.1. Introduction

Section 1.2, “Quick Start” will get you up and running on a single-node instance of HBase using the local filesystem. Chapter 2, Configuration describes setup of HBase in distributed mode running on top of HDFS.

1.2. Quick Start

This guide describes setup of a standalone HBase instance that uses the local filesystem. It leads you through creating a table, inserting rows via the HBaseshell, and then cleaning up and shutting down your standalone HBase instance. The below exercise should take no more than ten minutes (not including download time).

Hadoop : HBase

HBase is the Hadoop database. Use it when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. HBase is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable: A Distributed Storage System for Structured by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Hadoop. HBase includes:

• Convenient base classes for backing Hadoop MapReduce jobs with HBase tables including cascading, hive and pig source and sink modules
• Query predicate push down via server side scan and get filters
• Optimizations for real time queries
• A Thrift gateway and a REST-ful Web service that supports XML, Protobuf, and binary data encoding options
• Extensible jruby-based (JIRB) shell
• Support for exporting metrics via the Hadoop metrics subsystem to files or Ganglia; or via JMX

1.2. Quick Start

This guide describes setup of a standalone HBase instance that uses the local filesystem. It leads you through creating a table, inserting rows via the HBase shell, and then cleaning up and shutting down your standalone HBase instance. The below exercise should take no more than ten minutes (not including download time).

1.2.1. Download and unpack the latest stable release.

Choose a download site from this list of Apache Download Mirrors. Click on suggested top link. This will take you to a mirror of HBase Releases. Click on the folder named stable and then download the file that ends in .tar.gz to your local filesystem; e.g. hbase-0.93-SNAPSHOT.tar.gz.
Decompress and untar your download and then change into the unpacked directory.

$ tar xfz hbase-0.93-SNAPSHOT.tar.gz
$ cd hbase-0.93-SNAPSHOT

At this point, you are ready to start HBase. But before starting it, you might want to edit conf/hbase-site.xml and set the directory you want HBase to write to, hbase.rootdir.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///DIRECTORY/hbase</value>
  </property>
</configuration>

Replace DIRECTORY in the above with a path to a directory where you want HBase to store its data. By default, hbase.rootdir is set to /tmp/hbase-${user.name} which means you'll lose all your data whenever your server reboots (Most operating systems clear /tmp on restart).

1.2.2. Start HBase

Now start HBase:

$ ./bin/start-hbase.sh
starting Master, logging to logs/hbase-user-master-example.org.out

You should now have a running standalone HBase instance. In standalone mode, HBase runs all daemons in the the one JVM; i.e. both the HBase and ZooKeeper daemons. HBase logs can be found in the logs subdirectory. Check them out especially if HBase had trouble starting.

Is java installed?

All of the above presumes a 1.6 version of Oracle java is installed on your machine and available on your path; i.e. when you type java, you see output that describes the options the java program takes (HBase requires java 6). If this is not the case, HBase will not start. Install java, edit conf/hbase-env.sh, uncommenting the JAVA_HOME line pointing it to your java install. Then, retry the steps above.

1.2.3. Shell Exercises

Connect to your running HBase via the shell.

$ ./bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version: 0.90.0, r1001068, Fri Sep 24 13:55:42 PDT 2010

hbase(main):001:0> 

Type help and then <RETURN> to see a listing of shell commands and options. Browse at least the paragraphs at the end of the help emission for the gist of how variables and command arguments are entered into the HBase shell; in particular note how table names, rows, and columns, etc., must be quoted.
Create a table named test with a single column family named cf. Verify its creation by listing all tables and then insert some values.

hbase(main):003:0> create 'test', 'cf'
0 row(s) in 1.2200 seconds
hbase(main):003:0> list 'table'
test
1 row(s) in 0.0550 seconds
hbase(main):004:0> put 'test', 'row1', 'cf:a', 'value1'
0 row(s) in 0.0560 seconds
hbase(main):005:0> put 'test', 'row2', 'cf:b', 'value2'
0 row(s) in 0.0370 seconds
hbase(main):006:0> put 'test', 'row3', 'cf:c', 'value3'
0 row(s) in 0.0450 seconds

Above we inserted 3 values, one at a time. The first insert is at row1, column cf:a with a value of value1. Columns in HBase are comprised of a column family prefix -- cf in this example -- followed by a colon and then a column qualifier suffix (a in this case).
Verify the data insert.
Run a scan of the table by doing the following

hbase(main):007:0> scan 'test'
ROW        COLUMN+CELL
row1       column=cf:a, timestamp=1288380727188, value=value1
row2       column=cf:b, timestamp=1288380738440, value=value2
row3       column=cf:c, timestamp=1288380747365, value=value3
3 row(s) in 0.0590 seconds

Get a single row as follows

hbase(main):008:0> get 'test', 'row1'
COLUMN      CELL
cf:a        timestamp=1288380727188, value=value1
1 row(s) in 0.0400 seconds

Now, disable and drop your table. This will clean up all done above.

hbase(main):012:0> disable 'test'
0 row(s) in 1.0930 seconds
hbase(main):013:0> drop 'test'
0 row(s) in 0.0770 seconds 

Exit the shell by typing exit.

hbase(main):014:0> exit

1.2.4. Stopping HBase

Stop your hbase instance by running the stop script.

$ ./bin/stop-hbase.sh
stopping hbase...............