Nutch Crawl Command

• -dir : dir names the directory to put the crawl in.
• -threads : threads determines the number of threads that will fetch in parallel.
• -depth : depth indicates the link depth from the root page that should be crawled.
• -topN : N determines the maximum number of pages that will be retrieved at each level up to the depth.

  bin/nutch crawl urls -dir crawl -depth 5 -topN 50

Nutch Error: no agents listed in ‘http.agent.name’ property

Add the value to the file “nutch-site.xml”, “nutch-default.xml” under the folder: runtime/local, not the files under the root folder.

Run Nutch In Eclipse

Source  : http://wiki.apache.org/nutch/RunNutchInEclipse

Before you start

Setting up Nutch to run into Eclipse can be tricky, and most of the time you are much faster if you edit Nutch in Eclipse but run the scripts from the command line. However, it's very useful to be able to debug Nutch in Eclipse and is also extremely useful when applying and testing patches as it enables you to see them working in a larger context. This being said, you will still benefit greatly by looking at the hadoop.log output.

This tutorial covers a fully internal Eclipse/Nutch set up, using only Eclipse tools and associated plugins.

Prerequsites

• Grab the newest version of Eclipse availble here.
• All of the following should be available from the Eclipse Marketplace. However if not, you can download them throughout Eclipse as follows.
• Once you've set up Eclipse, download Subclipse as per here. N.B. If you experience an error with the 1.8.x release, try 1.6.x. This tends to solve compatibility problems.
• Grab IvyDE plugin for Eclipse as here.
• Grab m2e plugin for Eclipse here

Contents

1. RunNutchInEclipse
   1. Tested with
   2. Before you start
   3. Prerequsites
   4. Steps
      1. Install Nutch
      2. Establish the Eclipse environment for Nutch
      3. Configure Nutch
      4. Build Nutch
   5. Create Eclipse launcher
   6. Debug Nutch in Eclipse
   7. If things do not work...
      1. eclipse: Cannot create project content in workspace
      2. plugin directory not found
      3. No plugins loaded during unit tests in Eclipse
      4. classNotFound
      5. debugging Hadoop classes

NutchTutorial

Source : http://wiki.apache.org/nutch/NutchTutorial

Introduction

Apache Nutch is an open source Web crawler written in Java. By using it, we can find Web page hyperlinks in an automated manner, reduce lots of maintenance work, for example checking broken links, and create a copy of all the visited pages for searching over. That’s where Apache Solr comes in. Solr is an open source full text search framework, with Solr we can search the visited pages from Nutch. Luckily, integration between Nutch and Solr is pretty straightforward as explained below.

Apache Nutch release 1.3 has Solr integration embedded, greatly simplifying Nutch-Solr integration. It also removes the legacy dependence upon both Apache Tomcat for running the old Nutch Web Application and upon Apache Lucene for indexing. Just download a 1.3 binary release from here.

Table of Contents

Contents

1. Introduction
2. Table of Contents
3. Steps
4. 1 Setup Nutch from binary distribution
5. 2. Verify your Nutch installation
6. 3. Crawl your first website
   1. 3.1 Using the Crawl Command
   2. 3.2 Using Individual Commands for Whole-Web Crawling
7. 4. Setup Solr for search
8. 5. Verify Solr installation
9. 6. Integrate Solr with Nutch