Thursday, March 14, 2013

Using C++ or C to interact with hadoop

Are you a C++ or c programmer and you are not willing to write java code to interact with Hadoop/HDFS ha? Ok you have an option that is : llbhdfs native library that enables you to write programs in c or cpp to interact with Hadoop.

Current Hadoop distributions contain the pre-compiled libhdfs libraries for 32-bit and 64-bit Linux operating systems. You may have to download the Hadoop standard distribution and compile the libhdfs library from the source code, if your operating system is not compatible with the pre-compiled libraries.

For more information read following:

http://wiki.apache.org/hadoop/MountableHDFS

https://ccp.cloudera.com/display/CDHDOC/Mountable+HDFS

http://xmodulo.com/2012/06/how-to-mount-hdfs-using-fuse.html

Writing code in cpp or c Follows:

#include "hdfs.h"
int main(intargc, char **argv) {
hdfsFS fs =hdfsConnect( "NAMENODE_HOSTNAME,PORT);

if (!fs) {
fprintf(stderr, "
Cannot connect to HDFS.\n");
exit(-1);
}
char* readFile = "
cppcexample.file";
char* message = "
HDFS C API!!!";
int size = strlen(message);
int exists = hdfsExists(fs, readFile);
if (exists > -1) {
fprintf(stdout, "
File %s exists!\n", readFile);
}else{
// Create and open file for writing
hdfsFile outFile = hdfsOpenFile(fs, readFile, O_WRONLY|O_CREAT,
0, 0, 0);
if (!outFile) {
fprintf(stderr, "
Failed to open %s for writing!\n", readFile);
exit(-2);
}
// write to file
hdfsWrite(fs, outFile, (void*)message, size);
hdfsCloseFile(fs, outFile);
}
// Open file for reading
hdfsFile inFile = hdfsOpenFile(fs, readFile, O_RDONLY, 0, 0, 0);
if (!inFile) {
fprintf(stderr, "
Failed to open %s for reading!\n", readFile);
exit(-2);
}
char* data = malloc(sizeof(char) * size);
// Read from file.
tSize readSize = hdfsRead(fs, inFile, (void*)data, size);
fprintf(stdout, "
%s\n", data);
free(data);
hdfsCloseFile(fs, inFile);
hdfsDisconnect(fs);
return 0;
}

6 comments:

  1. Hi,
    I am doing a project to connect c program with hdfs.
    i downloaded hadoop r1.2.1.
    I tried the above code but when i executed it is showing as"cannot find hdfs.h"
    How to solve this?

    ReplyDelete
  2. Download this header file

    https://github.com/facebookarchive/hadoop-20/blob/master/src/c%2B%2B/libhdfs/hdfs.h

    ReplyDelete

  3. Thanks for this blog. provided great information. All the details are explained clearly with the great explanation.
    hadoop training in chennai

    ReplyDelete

  4. Thanks for this blog. provided great information. All the details are explained clearly with the great explanation.
    hadoop training in chennai

    ReplyDelete

  5. I really appreciate your post, and you explain each and every point very well. Thanks for sharing this information.
    C Sharp training and placement

    ReplyDelete

  6. I really appreciate your post, and you explain each and every point very well. Thanks for sharing this information.
    C Sharp training and placement

    ReplyDelete

Thank you for Commenting Will reply soon ......

Featured Posts

#Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc

 #Linux Commands Unveiled: #date, #uname, #hostname, #hostid, #arch, #nproc Linux is an open-source operating system that is loved by millio...