Incriment variable in linux shell

Following are the listed methods by which we can increment the variables in shell script in looping statements:

• j=$((i++))

• j=$(( i + 1 ))

• j=`expr $i + 1`

Happy scripting :)

Get file permission in numeric form in linux

We can use the following command to get the file permission in numeric form instead of string from.

stat -c "%a %n" *

Where 

-c   --> specify custom format

%a --> Access rights in octal format

%n --> file name

MySQL: Replication - Skip sql statement from execution

In any MySQL Replication set up, some or the other time we come across errors on the Slave due to which, replication stops.  

If the error is due to manual intervention on replication, we may have to skip the current SQL transaction on Slave which caused the error.

In the case, below is the command to be executed:

SET GLOBAL sql_slave_skip_counter=1;
START SLAVE;


You can increase the counter value if you want to skip multiple consecutive transactions.

Prevent accidental data loss in Hadoop

Some time you have accidentally delete some file which you are not suppose to do. So what will you do in that case? 

The option with you is to enable trash (Recycle bin) for hadoop, and define fs.trash.interval 

to one day with 1440 Minutes or more or less according to the cost of data you host in hadoop.

You can do it in following way :

Open core-site.xml and paste following parameter 

  

Display two files side by side in linux

Have you ever needed to display cat content of two files side by side on terminal, if yes you are right place :)

You have following options :

Starting/Stoping Hadoop and HBase in order

Hadoop Startup Process

Namenode  -->  Datanodes  -->  SecondaryNamenode  -->  JobTracker  -->  TaskTrackers

Hadoop Shutdown Process
JobTracker  -->  TaskTracker  --> Namenode  -->  Datanodes  --> SecondaryNamenode

HBase Startup Process

Zookeepers  -->  HMaster  -->  RegionServers

HBase Shutdown Process

RegionServers  -->  HMaster  -->  Zookeepers

Too many fetch-failures, hadoop mapreduce job, hadoop mapreduce Too many fetch-failures

This may happen due to following reasons:

1. Wrong DNS Entry, Hosts file entries, 

Description:

Nodes are not able to communicate with each other, so check if nodes where this error comes you can ping or  nslookup to that node

2. Tasktracker http thread

Description:

Check of the value of this tag in mapreduce-site.xml, if it is lower make it somewhat higer around 100.

Get Directory path from a file path/Get file name from a filepath in shell Linux/Unix

Get Directory path from a file path.

Command : 

dirname /path/path1/filename

Output :

 /path/path

Get filename from a filepath

Command:

basename /path/path/filename

Output:

filaname

Getting thread count of a process using PID in Linux

Using following command we can get the thread count of a process in Linux, using PID of the process.

ps -o thcoung -p <PID>

If we need some more detail like process type, pid, user under which the process is running we can use following command :

ps -o user,comm,pid,thcount -p <PID>

Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'xxxxxx' doesn't exist

FAILED: Error in metadata: MetaException(message:Got exception: javax.jdo.JDODataStoreException Exception thrown obtaining schema column information from datastore)
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: javax.jdo.JDODataStoreException Exception thrown obtaining schema column information from datastore

Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'xxxxxx' doesn't exist

Follow following to solve this problem.