Nutch Error: no agents listed in ‘http.agent.name’ property

Add the value to the file “nutch-site.xml”, “nutch-default.xml” under the folder: runtime/local, not the files under the root folder.

Run Nutch In Eclipse

Source  : http://wiki.apache.org/nutch/RunNutchInEclipse

Before you start

Setting up Nutch to run into Eclipse can be tricky, and most of the time you are much faster if you edit Nutch in Eclipse but run the scripts from the command line. However, it's very useful to be able to debug Nutch in Eclipse and is also extremely useful when applying and testing patches as it enables you to see them working in a larger context. This being said, you will still benefit greatly by looking at the hadoop.log output.

This tutorial covers a fully internal Eclipse/Nutch set up, using only Eclipse tools and associated plugins.

Prerequsites

• Grab the newest version of Eclipse availble here.
• All of the following should be available from the Eclipse Marketplace. However if not, you can download them throughout Eclipse as follows.
• Once you've set up Eclipse, download Subclipse as per here. N.B. If you experience an error with the 1.8.x release, try 1.6.x. This tends to solve compatibility problems.
• Grab IvyDE plugin for Eclipse as here.
• Grab m2e plugin for Eclipse here

Contents

1. RunNutchInEclipse
   1. Tested with
   2. Before you start
   3. Prerequsites
   4. Steps
      1. Install Nutch
      2. Establish the Eclipse environment for Nutch
      3. Configure Nutch
      4. Build Nutch
   5. Create Eclipse launcher
   6. Debug Nutch in Eclipse
   7. If things do not work...
      1. eclipse: Cannot create project content in workspace
      2. plugin directory not found
      3. No plugins loaded during unit tests in Eclipse
      4. classNotFound
      5. debugging Hadoop classes

NutchTutorial

Source : http://wiki.apache.org/nutch/NutchTutorial

Introduction

Apache Nutch is an open source Web crawler written in Java. By using it, we can find Web page hyperlinks in an automated manner, reduce lots of maintenance work, for example checking broken links, and create a copy of all the visited pages for searching over. That’s where Apache Solr comes in. Solr is an open source full text search framework, with Solr we can search the visited pages from Nutch. Luckily, integration between Nutch and Solr is pretty straightforward as explained below.

Apache Nutch release 1.3 has Solr integration embedded, greatly simplifying Nutch-Solr integration. It also removes the legacy dependence upon both Apache Tomcat for running the old Nutch Web Application and upon Apache Lucene for indexing. Just download a 1.3 binary release from here.

Table of Contents

Contents

1. Introduction
2. Table of Contents
3. Steps
4. 1 Setup Nutch from binary distribution
5. 2. Verify your Nutch installation
6. 3. Crawl your first website
   1. 3.1 Using the Crawl Command
   2. 3.2 Using Individual Commands for Whole-Web Crawling
7. 4. Setup Solr for search
8. 5. Verify Solr installation
9. 6. Integrate Solr with Nutch

Timer jobs in sharepoint

Manage timer jobs

You can check the status of a timer job and edit the timer job definition.

For the general administration of all jobs, the SharePoint Central Administration Web site has a Timer Job Status page and a Job Definitions page. You can find these pages in Central Administration, on the Monitoring page, in theTimer Jobs section.

From the View menu, you can filter the timer jobs at the following levels:

• All   Displays all timer jobs for the farm.
• Service   Displays all the timer jobs for a particular service. If you select this command, use the Service menu to select the service by which you want to filter the listed jobs.
• Web Application   Displays all the timer jobs for a Web application. If you select this option, use the Web Application menu to select the Web application by which you want to filter the listed jobs.
• Server   Displays all the timer jobs for the specified server. If you select this command, use the Server menu to select the server by which you want to filter the listed jobs.
• Job Definition   Displays all the timer jobs for the specified job definition. On the Timer Job Status page, use the Job Definition menu to select the job definition by which you want to filter the listed jobs.
• Failed Jobs   Displays all the timer jobs on the farm that have failed to finish.

The SharePoint 2010 Timer service (SPTimerv4) is based on the Gregorian calendar for scheduling. For every job that you schedule, you specify when the timer job will run, specified in a 24-hour time format. You must specify the time in local time instead of as an offset from Coordinated Universal Time (UTC). The time is stored in that format. Daily, weekly, and monthly schedules also include a window of execution. The timer service will select a random time within this interval to start executing the job on each applicable server. This feature is appropriate for high-load jobs that run on multiple servers on the farm. Running this kind of job on all the servers at the same time might place an unreasonable load on the farm. Timer job schedules can be specified by using Windows PowerShell. For more information, see Timer jobs cmdlets (SharePoint Server 2010).

Default timer jobs

Debugging and Logging Capabilities in SharePoint 2010

Microsoft SharePoint 2010 is a very extensible product that developers can customize to suit various business needs. Because SharePoint is based on Microsoft ASP.NET, it provides developers with the same debugging and logging options that are available to traditional ASP.NET websites. In addition to what ASP.NET provides, SharePoint 2010 introduces additional options to assist developers in monitoring and troubleshooting applications in the areas of debugging and logging.

This article touches on various debugging and logging points and techniques that developers can use in their custom SharePoint 2010 applications.

What's New in SharePoint 2010 for Logging and Debugging

The best applications contain a good measure of logging to assist developers in tracking down errors when things do not go as they were designed or when unexpected things occur. Microsoft Office SharePoint Server 2007 offered developers all the tools that were available in ASP.NET, such as the debug and trace log. Developers could also write to the SharePoint log, known as the Unified Logging Service (ULS), but it required some extra work to write directly to the log files.

Microsoft added more logging and debugging capabilities to SharePoint 2010 to help developers include additional monitoring and troubleshooting to their custom applications. This article addresses the following new areas in SharePoint 2010:

• Correlation tokens
  
• Logging database
  
• Custom error pages
  
• Developer Dashboard

Adding a Custom Action to a SharePoint 2010 Display Form

It is common when customizing SharePoint 2010 to want to extend the user interface. Fortunately, this is easy to do by using the ribbon menu. You can add new functionality to the SharePoint 2010 ribbon as custom actions. Custom actions enable you to expand or extend the standard behavior of SharePoint 2010 core components such as using a custom action to email the details of a list item.

This topic describes two approaches for adding a custom action to a display form of SharePoint 2010 list items. The custom action is added to the ribbon as a button.

You have two options to add custom actions to a display form:

• Adding a custom action through SharePoint 2010 features. SharePoint features enable you to easily deploy customizations. This is the preferred method when you need to package the customizations and deploy them to different servers.
  
• Adding a custom action by using Microsoft SharePoint Designer 2010. SharePoint Designer helps create rapid, no-code customizations on the local server.
  

Custom Action Elements

 

Creating a Custom Web Part Editor in SharePoint 2010

The Microsoft SharePoint 2010 Web Part framework enables users to configure a Web Part through Web Part properties. A default user interface (UI) is given for each Web Part property that is exposed to the user in the Edit Tool pane of the Web Part. For example, a property of type String is rendered as a text box, and an Enum is rendered as a drop-down list box.

This default rendering by the framework does not allow for much customization because the user cannot get a handle to the controls that are rendered. For example, it has the following limitations:

• Control over UI: Numeric and string types are rendered as text boxes, enumeration types are rendered as a drop-down list box, and Boolean types are rendered as check boxes. So, you cannot show an enumeration type as radio buttons instead of a drop-down list box.
  
• Events: Events that are raised by the controls cannot be handled; for example, you cannot enable or disable a drop-down list based on selecting a check box.
  
• Validation: If user input has to be validated, the framework does not provide a way to easily attach a Microsoft ASP.NET validation control to the input control that is generated. For example, you cannot validate a string input for a valid email address through an ASP.NET RegEx validation control.
  

You can overcome these limitations by using custom editor parts. With custom editor parts, you can offer users functionality that can be achieved through ASP.NET user controls. A Web Part can implement custom editor parts that can be loaded when the Web Part is in edit mode, expose a custom UI, handle events, and validate inputs. A Web Part can have more than one custom editor part associated with it.

This article describes the SharePoint 2010 Web Part framework in detail and provides a code sample to show how to create a custom Web Part editor in SharePoint 2010. First, we examine the framework that enables custom editor parts.

C API to HDFS: libhdfs

#include "hdfs.h"

int main(int argc, char **argv) {

    hdfsFS fs = hdfsConnect("default", 0);
    const char* writePath = "/tmp/testfile.txt";
    hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0);
    if(!writeFile) {
          fprintf(stderr, "Failed to open %s for writing!\n", writePath);
          exit(-1);
    }
    char* buffer = "Hello, World!";
    tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1);
    if (hdfsFlush(fs, writeFile)) {
           fprintf(stderr, "Failed to 'flush' %s\n", writePath);
          exit(-1);
    }
   hdfsCloseFile(fs, writeFile);
}

HDFS Permissions

Overview

The Hadoop Distributed File System (HDFS) implements a permissions model for files and directories that shares much of the POSIX model. Each file and directory is associated with an owner and a group. The file or directory has separate permissions for the user that is the owner, for other users that are members of the group, and for all other users. For files, the r permission is required to read the file, and the wpermission is required to write or append to the file. For directories, the r permission is required to list the contents of the directory, the w permission is required to create or delete files or directories, and the xpermission is required to access a child of the directory. In contrast to the POSIX model, there are no sticky, setuid or setgid bits for files as there is no notion of executable files. For directories, there no sticky,setuid or setgid bits directory as a simplification. Collectively, the permissions of a file or directory are its mode. In general, Unix customs for representing and displaying modes will be used, including the use of octal numbers in this description. When a file or directory is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule).

Each client process that accesses HDFS has a two-part identity composed of the user name, and groups list. Whenever HDFS must do a permissions check for a file or directory foo accessed by a client process,

• If the user name matches the owner of foo, then the owner permissions are tested;
• Else if the group of foo matches any of member of the groups list, then the group permissions are tested;
• Otherwise the other permissions of foo are tested.

If a permissions check fails, the client operation fails.

User Identity

In this release of Hadoop the identity of a client process is just whatever the host operating system says it is. For Unix-like systems,

• The user name is the equivalent of `whoami`;
• The group list is the equivalent of `bash -c groups`.

In the future there will be other ways of establishing user identity (think Kerberos, LDAP, and others). There is no expectation that this first method is secure in protecting one user from impersonating another. This user identity mechanism combined with the permissions model allows a cooperative community to share file system resources in an organized fashion.

In any case, the user identity mechanism is extrinsic to HDFS itself. There is no provision within HDFS for creating user identities, establishing groups, or processing user credentials.

HBase MapReduce Read/Write Example

Configuration config = HBaseConfiguration.create();
Job job = new Job(config,"ExampleReadWrite");
job.setJarByClass(MyReadWriteJob.class);    // class that contains mapper
                       
Scan scan = new Scan();
scan.setCaching(500);        // 1 is the default in Scan, which will be bad for MapReduce jobs
scan.setCacheBlocks(false);  // don't set to true for MR jobs
// set other scan attrs
           

HBase MapReduce Read Example

Configuration config = HBaseConfiguration.create();
Job job = new Job(config, "ExampleRead");
job.setJarByClass(MyReadJob.class);     // class that contains mapper
   
Scan scan = new Scan();
scan.setCaching(500);        // 1 is the default in Scan, which will be bad for MapReduce jobs
scan.setCacheBlocks(false);  // don't set to true for MR jobs
// set other scan attrs