Recursively list all files in a directory Linux

find -follow -type f   à find all files recursively type file

ll –R  à long list recursively

ls –LR à List all recursively

tree –l  à tree structure  

find /dir -type f -follow -print| xargs ls –l   à file and list all files

Find and move list of file spread over diffrent location on a linux machine

#----------------------------------------------------------------#
#  USES : sh fileFinderMover.sh <filewithfilenames> <targetpath> #
#----------------------------------------------------------------#

#!bin/sh
for i in `cat $1`
do
path=`locate $i`
mv $path $2
echo $path
done


This script will seach for the file which give as list and will move to target location.

hadoop oiv : hadoop offline image viewer, get a list of all files on hadoop

Replace a word in file Linux Shell

sed -i 's/<word to replace>/<new word>/g'<filenametobereplacedin>

Eg:

sed -i 's/shashwat/shriparv/g'shashwat.txt

Better way to get a list of all files on Hadoop, using shell.

Hadoop provides an option OIV  that is offline image viewer, which can read the Hadoop image file and output it to a output file in human readable format.

Syntax and uses:

bin/hadoop  oiv –i <Hadoop image file name> -o <output file in human readable form>

Options with this command:

There is an additional option to output file format that is defined using switch –p <format>, the formats can be: -->

 

List Files from hdfs/Hadoop Recursively using java

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * @author Shashwat Shriparv
 * @email  dwivedishashwat@gmail.com
 * @web    helpmetocode.blogspot.com
 */
public class RecursivelyPrintFilesOnHDFS {

    public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
        printFilesRecursively("hdfs://master1:9000/");
    }

    public static void printFilesRecursively(String Url) throws IOException {
        try {

Write file to HDFS/Hadoop Read File From HDFS/Hadoop Using Java

import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * @author    Shashwat Shriparv
 * @email     dwivedishashwat@gmail.com
 * @Web       helpmetocode.blogspot.com
 */
public class WritetoHDFSReadFromHDFSWritToLocal {
    private static byte[] buffer;
    private static int bytesRead;

    public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
      

Insert string after each N lines in a file

We can do this as follows:

awk '1;!(NR%<Number after which the line has to be insserted>){print "String to be inserted";}' origionalfiletoprocess >outfilewithinsertedstring

Eg:

awk '1;!(NR%100){print "Shashwat Shriparv";}' filecontainingtxt>outputfilewithnewinsertedlines

This command will read filecontainingtxt and will insert string Shashwat Shriparv after 100 lines the the output fill will be outputfilewithnewinsertedlines

Big data, map reduce and beyond

Big data, map reduce and beyond from datasalt

WebHDFS REST API

The HTTP REST API supports the complete FileSystem interface for HDFS.

Operations

• HTTP GET
  • OPEN (see FileSystem.open)
  • GETFILESTATUS (see FileSystem.getFileStatus)
  • LISTSTATUS (see FileSystem.listStatus)
  • GETCONTENTSUMMARY (see FileSystem.getContentSummary)
  • GETFILECHECKSUM (see FileSystem.getFileChecksum)
  • GETHOMEDIRECTORY (see FileSystem.getHomeDirectory)
  • GETDELEGATIONTOKEN (see FileSystem.getDelegationToken)
• HTTP PUT
  • CREATE (see FileSystem.create)
  • MKDIRS (see FileSystem.mkdirs)
  • RENAME (see FileSystem.rename)
  • SETREPLICATION (see FileSystem.setReplication)
  • SETOWNER (see FileSystem.setOwner)
  • SETPERMISSION (see FileSystem.setPermission)
  • SETTIMES (see FileSystem.setTimes)
  • RENEWDELEGATIONTOKEN (see DistributedFileSystem.renewDelegationToken)
  • CANCELDELEGATIONTOKEN (see DistributedFileSystem.cancelDelegationToken)
• HTTP POST
  • APPEND (see FileSystem.append)
• HTTP DELETE
  • DELETE (see FileSystem.delete)

For More Please Visit WebHDFS