FAILED FSError: java.io.IOException: No Space left on device

 

FAILED FSError: java.io.IOException: No Space left on device

org.apache.hadoop.util.DiskChecker$DiskErrorException : Could not find any valid local directory for taskTracker/jobcache

When you run hive queries and you queries fails, just go the the job tracker of Hadoop : mostly on 50030 port on your Hadoop nodes, and check the failed or killed jobs, you may find these errors.

The reason behind will be following:

Shell script: Using FOR loop

Here's a simple shell script to print numbers using for loop.

Just got through:

# cat test.sh

for ((i=0;i<=10;i++))
do

 echo $i

done


Output:

Do you need easier way to view or manipulate the data saved in the - HBase

Here i found two projects that will connect to HBase and will let you to view and manipulate data in HBase. Just though of sharing with you all.

Automatically Finding files of specific size of type or modified date and adding to tar

Refer following command for your specific need:

#All files of size greater thab 1Gb

find / -type f -size +1048576 | xargs tar -czf myfile.tgz

#All files modified one day before

find / -type f -mtime -1 | xargs tar -czf myfile.tgz

#All files modified one day before and size more than 1Gb

find / -type f -mtime -1 -size +1048576 | xargs tar -czf myfile.tgz

------------------------ OR ------------------------------------

Password-less authentication userful for Hadoop Administrators and linux users

Although Hadoop never requires any password-less authentication to communicate between nodes, but from Hadoop administrator perspective it provide a great flexibility while managing multiple nodes together.

a. Generate the SSH private/public key pair for hadoop user in Namenode

i. ssh-keygen –t rsa

Linux : Prefix a string to all rows in a file - sed

Consider, you have a file like below:

# cat test.txt
12324
234523
3431
343
34345
3434
43234

Now, if you want to prefix a particular string to each line in the file, how will you do it? This is a common challenge for DBA/Developers working with Linux environment.

Linux: Comparing 2 files - 'comm'

Many a times, i encountered the situation wherein I would be having 2 files with some values in each. And I need to get common records in both the files and also, unique records for each files.

So, this is how  I do - one of the easiest way.

Below are the files to be compared:

MySQL Replication: Exclude an SQL from replicating

If you're using MySQL Replication, in rare cases you may require to skip a particular SQL statement from replicating - which is to be executed on Master and should not be on the Slave. Though, its not a good practice.

However, here's how to do it. 

MySQL has a variable sql_log_bin, by default its value will be 'ON' or '1', when we enable replication. That means, it replicates all the SQL statements from Master to Slave.
So, if we want to disable this for a particular SQL, do as below:

SET SESSION sql_log_bin=0; sql statement;SET SESSION sql_log_bin=1;


Example:

 
SET SESSION sql_log_bin=0; DELETE FROM tab1 WHERE col1='NULL'; 
SET SESSION sql_log_bin=1;

 However, after MySQL 5.5.5 version, no need to use SESSION. By default its a SESSION variable and it won't affect other user's transactions. 

Note: Executing this on Master would cause data inconsistency. Don't try this unless you are sure about what you are doing!

Top Command Options for Admnistrators

There are lot of option with top command but some parameters which i found very useful are following :


1. Type top on terminal you will get interface something like :




So mostly What do we need to see??

Memory wise sorted processes
CPU utilization wise sorted process and 
The process name which is using the memory and the CPU Right?

Linux: Add numbers in a column in a file

In windows we use Excel to add all the numbers in a column. But how about Unix/Linux?
We encountered such an requirement and we got the answer too.
Actually solution in much more simpler than the problem..!

Mission accomplished by 'awk' command. Its merely a very powerful and useful command.
Here is an example for it:

Convert image to byte array and byte array to image file

This code will convert an image file to byte array and will write it into a binary file, and will read the binary file and converti it to image file.

import java.awt.image.BufferedImage;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.BufferedInputStream;
import javax.imageio.ImageIO;


public class conveter {

 /**
  * @param args
  */
 public static void main(String[] args) {
  try {
    
   byte[] imageInByte;
   BufferedImage originalImage = ImageIO.read(new File(
     "Chrysanthemum.jpg"));