List top 10/n biggest/smallest files on hadoop (size wise)

List top 10 biggest files in a directory on hadoop:
hadoop dfs -du /testfiles/hd|sort -g -r|head -n  <N>  {N here is the top number of file you want to list}
      
hadoop dfs -du /testfiles/hd|sort -g -r|head -n  10

List top 10 biggest file on hadoop(Recursively) :


hadoop dfs -lsr /|awk '{print $5 "\t\t" $8}'|sort -n -r|head -n <N> {N here is the top numbers of files you want to list}
hadoop dfs -lsr /|awk '{print $5 "\t\t" $8}'|sort -n -r|head -n 10

List top 10 smallest files in a directory on hadoop:
hadoop dfs -du /testfiles/hd|sort -g -r|tail -n  <N>  {N here is the top number of file you want to list}
      
hadoop dfs -du /testfiles/hd|sort -g -r|tail -n  10

List top 10 smallest file on hadoop(Recursively) :


hadoop dfs -lsr /|awk '{print $5 "\t\t" $8}'|sort -n -r|tail -n <N> {N here is the top numbers of files you want to list}
hadoop dfs -lsr /|awk '{print $5 "\t\t" $8}'|sort -n -r|tail -n 10

Hadoop version support matrix, hadoop Hbase Version compatibility

	HBase-0.92.x	HBase-0.94.x	HBase-0.95
Hadoop-0.20.205	S	X	X
Hadoop-0.22.x	S	X	X
Hadoop-1.0.0-1.0.2	S	S	X
Hadoop-1.0.3+	S	S	S
Hadoop-1.1.x	NT	S	S
Hadoop-0.23.x	X	S	NT
Hadoop-2.x	X	S	S
HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.

Where

S = supported and tested,
X = not supported,
NT = it should run, but not tested enough.

Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its lib directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is critical that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up.

Keep your locate database updated in linux

We may use find command to find a file in linux but it is a bit slower than Locate, 

locate is a command that help us to find a file efficiently and a bit faster, 

it maintains a database called

"/var/lib/mlocate/mlocate.db" 

and also has a utility called "updatedb" which keeps this database updated with the 

file which

are there in the system. so when we run this command this command runs it update 

the database

and updates this database with newly file added.  This database is kept by linux system to 

keep most updated file list.

if you want a file not to be modified by anyone even the root user

Suppose you have a file which you never want any one to delete, modify, move here is the option you have......

This is called an immutable file which is supported in linux ext2/ext3

which you can do as follow:

suppose you want to create a file called configuration which you never wanted to be modified move or removed the what you can do is follows:

chattr +i configuration

after doing that this file configuration cant be deleted,moved or changed

if you want it to simple file again you  can do

chattr -i configuration

Recursively list all files in a directory Linux

find -follow -type f   à find all files recursively type file

ll –R  à long list recursively

ls –LR à List all recursively

tree –l  à tree structure  

find /dir -type f -follow -print| xargs ls –l   à file and list all files

Find and move list of file spread over diffrent location on a linux machine

#----------------------------------------------------------------#
#  USES : sh fileFinderMover.sh <filewithfilenames> <targetpath> #
#----------------------------------------------------------------#

#!bin/sh
for i in `cat $1`
do
path=`locate $i`
mv $path $2
echo $path
done


This script will seach for the file which give as list and will move to target location.

hadoop oiv : hadoop offline image viewer, get a list of all files on hadoop

Replace a word in file Linux Shell

sed -i 's/<word to replace>/<new word>/g'<filenametobereplacedin>

Eg:

sed -i 's/shashwat/shriparv/g'shashwat.txt

Better way to get a list of all files on Hadoop, using shell.

Hadoop provides an option OIV  that is offline image viewer, which can read the Hadoop image file and output it to a output file in human readable format.

Syntax and uses:

bin/hadoop  oiv –i <Hadoop image file name> -o <output file in human readable form>

Options with this command:

There is an additional option to output file format that is defined using switch –p <format>, the formats can be: -->

 

List Files from hdfs/Hadoop Recursively using java

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * @author Shashwat Shriparv
 * @email  dwivedishashwat@gmail.com
 * @web    helpmetocode.blogspot.com
 */
public class RecursivelyPrintFilesOnHDFS {

    public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
        printFilesRecursively("hdfs://master1:9000/");
    }

    public static void printFilesRecursively(String Url) throws IOException {
        try {