Package org.apache.nutch.hostdb

Class UpdateHostDbMapper

java.lang.Object

org.apache.hadoop.mapreduce.Mapper<Text,Writable,Text,NutchWritable>

org.apache.nutch.hostdb.UpdateHostDbMapper

public class UpdateHostDbMapper
extends Mapper<Text,Writable,Text,NutchWritable>

Mapper ingesting HostDB and CrawlDB entries. Additionally it can also read host score info from a plain text key/value file generated by the Webgraph's NodeDumper tool.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper

Mapper.Context

Field Summary

Fields

Modifier and Type	Field	Description
protected String[]	args
protected String	buffer
protected CrawlDatum	crawlDatum
protected ErrorTracker	errorTracker
protected boolean	filter
protected Counter	filteredRecordsCounter
protected URLFilters	filters
protected Text	host
protected HostDatum	hostDatum
protected boolean	normalize
protected URLNormalizers	normalizers
protected boolean	readingCrawlDb
protected String	reprUrl

Constructor Summary

Constructors

Constructor	Description
UpdateHostDbMapper()

Method Summary

Modifier and Type	Method	Description
protected String	filterNormalize(String hostName)	Filters and or normalizes the input hostname by applying the configured URL filters and normalizers the URL "http://hostname/".
void	map(Text key, Writable value, Mapper<Text,Writable,Text,NutchWritable>.Context context)	Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file.
void	setup(Mapper<Text,Writable,Text,NutchWritable>.Context context)

Methods inherited from class org.apache.hadoop.mapreduce.Mapper

cleanup, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

host

protected Text host

hostDatum

protected HostDatum hostDatum

crawlDatum

protected CrawlDatum crawlDatum

reprUrl

protected String reprUrl

buffer

protected String buffer

args

protected String[] args

filter

protected boolean filter

normalize

protected boolean normalize

readingCrawlDb

protected boolean readingCrawlDb

filters

protected URLFilters filters

normalizers

protected URLNormalizers normalizers

filteredRecordsCounter

protected Counter filteredRecordsCounter

errorTracker

protected ErrorTracker errorTracker

Constructor Details

UpdateHostDbMapper

public UpdateHostDbMapper()

Method Details

setup

public void setup(Mapper<Text,Writable,Text,NutchWritable>.Context context)

Overrides:

setup in class Mapper<Text,Writable,Text,NutchWritable>

filterNormalize

protected String filterNormalize(String hostName)

Filters and or normalizes the input hostname by applying the configured URL filters and normalizers the URL "http://hostname/".

Parameters:

hostName - the input hostname

Returns:

the normalized hostname, or null if the URL is excluded by URL filters or failed to be normalized converted

map

public void map(Text key,
 Writable value,
 Mapper<Text,Writable,Text,NutchWritable>.Context context)
         throws IOException,
InterruptedException

Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file. Statistics and scores are passed on.

Overrides:

map in class Mapper<Text,Writable,Text,NutchWritable>

Parameters:

key - record Text key

value - associated Writable object

context - Reducer.Context for writing custom counters and output.

Throws:

IOException

InterruptedException