Package org.apache.nutch.protocol

Interface Protocol

All Superinterfaces:

Configurable, Pluggable

All Known Implementing Classes:

File, Ftp, Http, Http, Http, Http, Http, HttpBase, OkHttp

public interface Protocol
extends Pluggable, Configurable

A retriever of url content. Implemented by protocol extensions.

Field Summary

Fields

Modifier and Type	Field	Description
static String	X_POINT_ID	The name of the extension point.

Method Summary

Modifier and Type	Method	Description
ProtocolOutput	getProtocolOutput(Text url, CrawlDatum datum)	Get the ProtocolOutput for a given url and crawldatum
crawlercommons.robots.BaseRobotRules	getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)	Retrieve robot rules applicable for this URL.

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

X_POINT_ID

static final String X_POINT_ID

The name of the extension point.

Method Detail

getProtocolOutput

ProtocolOutput getProtocolOutput(Text url,
                                 CrawlDatum datum)

Get the ProtocolOutput for a given url and crawldatum

Parameters:

url - canonical url

datum - associated CrawlDatum

Returns:

the ProtocolOutput

getRobotRules

crawlercommons.robots.BaseRobotRules getRobotRules(Text url,
                                                   CrawlDatum datum,
                                                   List<Content> robotsTxtContent)

Retrieve robot rules applicable for this URL.

Parameters:

url - URL to check

datum - page datum

robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.

Returns:

robot rules (specific for this URL or default), never null