Comparison between RDBMS and Map-Reduce

	RDBMS	Map-Reduce
Scalability	Scale UP	Scale Out
Data Size	GB	PB and more
Read/Write	Batch and Interactive	Batch
Update Type	Write many Read many	Write once Read many
Integrity	High	Low
Structure	Structural data/ Schema first Write later	Non Structural data/ Write first Schema later
Query	SQL	No SQL and SQL support too with add on tools
Response time	Faster for Less data/ slow once size increases	Faster for more data in comparison
Note: Slowly with the new sub projects being developed for Hadoop this gap is being filled up as people are developing an abstract layer on top of Map-Reduce and YARN frameworks which takes SQL and in turn convert it to Map-Reduce/Yarn.

Extract All Tar Files in a directory in Linux

This will first list the files containing extension tar.gz, and then awk will get the file names which is column 9 in

ls -lrth command, and

NF > 2 will remove the blank line and

tar -xvzf will extract files names contained in variable $i,

Like this we can experiment various operations like renaming all files with specific extension of so and can fiddled with to achieve various goals.


for i in `ls -lrth *.tar.gz |awk  'NF>2 {print $9}'`; do tar -xvzf $i; done

All about hadoop Balancer.

Hadoop Data Balancing

Hadoop Balancer:

This is tool provided to balance the disk uses throughout the Hadoop cluster. I may happen sometime that some of the nodes in the cluster becomes over utilized or underutilized, which occurs due to addition of new nodes where newly added nodes may be underutilized or if there are less number of nodes result in overutilization. We can run balancer from more than 1 machine in the cluster to increase the speed of balancing but it will increase bandwidth uses to very high.

This tool requires administrator right on the Hadoop cluster to run.

Syntax of the balancer:

bin/start-balancer.sh [-threshold <threshold>]

Where start-balancer.sh files resides in the bin directory of the Hadoop folder. And the threshold is the parameter which decides target of balance, this lies in fraction between 0,1 the default value is 10% if nothing is passed as the threshold value.

This process does the transferring of blocks between the nodes resulting network activity and if a production cluster must be used cautiously, as it result in some block missing error or late reply from the cluster.

This process can be stopped any time if required using following command:

WPF Architecture

WPF architecture is multilayered architecture.  It has three layers mainly Managed code, Unmanaged code and  Core Operating system. We can call these layers as set of assemblies that built up the entire framework.

The major components are below Presentation Framework, Presentation Core and Media Integration(Milcore)  are the major components of wpf architecture. 

 

Limit Disk uses in Datanodes Hadoop

There may in some scenario when disk attached to data node may go over utilize and you become unable to perform any operation on the data node due to no space left on the system. so we have a option of defining a limit of space which can be used by data node daemons.
Using following configuration:

<property>
 <name>dfs.datanode.du.reserved</name>
 <value>182400</value>
 <description>Reserved space in bytes per volume. This defines to leave this much space free for non dfs use.</description>
</property>

Reading sequence file/ compressed file / or TextRecordInputStream file from hadoop

As we know sequence file are binary files with key/value pair specially build for hadoop. If we want to read file on hadoop we have option of cat which will not show the output of sequence or compressed file correctly in this case we can use -text option with hadoop command which follows:

hadoop fs -text   <file path name/filename>

which will show they correct output of the sequence file content.

Features Supported by Hadoop Release Series

Comparison between Big Data and RDBMS

	RDBMS	Big Data
Data size	Gigabytes	Petabytes
Access	Interactive and batch	Batch
Updates	Read and write many times	Write once, read many times
Structure	Static schema	Dynamic schema
Integrity	High	Low
Scaling	Nonlinear	Linear

HBase Backup

HBase Backup:

We need to have backup of HBase table offline in some point of time, in spite of the fact that Hadoop and HBase provide replication and redundancy. For this we have some backup option in HBase. These are categorized in two ways:

Offline backup and

Online backup

Online Backup:

Again this is categorized in three ways

Replication: In this method you need to have a 2^nd cluster where you will keep your replication for the data from the 1^st cluster.

Hadoop/HBase Export command: which runs a map reduce job to copy table from one cluster to the same cluster or to other Hadoop cluster. This does not require any kind of downtime for backing/ exporting data.

In this method we need to export the data to the cluster and if we need to restore we need to restore it by Importing.

CopyTable: this is also online backup method which copies table from one cluster to another cluster or to the same cluster.

Offline Backup:

Distcp : this is a kind of file system backup, this copies a directory from HDFS to same cluster or to other cluster.

copyToLocal : this is less reliable way of copying directories from HDFS to local backup drive. If large amount of data is there then you need lot of Hadoop tune-up to copy successfully.

Offline Backup methods are full shutdown backup method, suppose you need to copy HBase you need to stop your HBase cluster, for a successful backup, as the files are being continuously moved, modified and changes while cluster is online, and copying in this scenario may fail.

Linux: Crontab - Brief

I always get confuse whenever I want to set a new cron job. The confuse is with regard to the options too be set!
For those who new to 'cron', its nothing but, an event scheduler in Linux. That means, you can schedule any script to run at any time you wanted to. Its just the system/server should be up and running!

cron job is specific to every user in Linux/Unix. So, one can't see other's cron unless the necessary privileges or sudo root access given.

Whatever, here is the options in cron:


To check cron jobs:

[root@localhost kiran]# crontab -l
no crontab for root

To set cron jobs:

[root@localhost kiran]# crontab -e

After adding, here is how it looks:

[root@localhost kiran]# crontab -l

##Script to test

00 */2 1-31 * 0,2,3   sh /home/kiran/test.sh >> /dev/null

Every Cron job should be given with 5 options:

• minute -> 0-59
• hour -> 0-23
• day of month -> 1-31
• month -> 1-12 
• day of week -> 0-7 (0 is Sunday )

In the above example:

00 -- 0th Minute

*/2 -- Every 2 hours

1-31 -- Every day (1 to 31)

* -- Every Month

0,2,3 -- Sunday,Tuesday,Wednesday

@@ the Techy Talks..