Infinite Programming Tips: Copy file from local disk to hdfs using java

Thursday, March 1, 2012

Copy file from local disk to hdfs using java

Local File ====== File Copy =======> hdfs file system

Create a project in eclipse or netbeans or any editor you like and add hadoop-core.jar and create a class named PutToHdfs and put this code

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * 
 * @author shashwat
 */
public class PutToHdfs {

    /**
     * @param args the command line arguments

this contains two variation of file copying to hdfs your accordingly as your requirement, explanation given in comment.
     */
    public static void main(String[] args) throws IOException, URISyntaxException {
        // TODO code application logic here
        //InetSocketAddress add = new InetSocketAddress("192.168.20.12", 9000);   //-------> Use this if you are using DistributedFileSystem Class(For hadoop configured as distributed)
        URI url=new URI("hdfs://192.168.0.1:9000"); //-------> (url where hdfs located-for detail look hadoop configuration )Use this if you are using FileSystem Class(For hadoop configured on a single system) 
        Configuration conf = new Configuration();
        //DistributedFileSystem ffs = new DistributedFileSystem(add, conf);
         FileSystem file1= FileSystem.get(url, conf);
        Path src = new Path("/sourcepath/filename.text");
        Path dst = new Path("/destinationhdfsfolder/");
        //fs.copyFromLocalFile(b1, b2, src, dst);      
        file1.copyFromLocalFile(src, dst);
    }
}