Package org.apache.nutch.scoring.similarity.util

Class LuceneTokenizer

java.lang.Object

org.apache.nutch.scoring.similarity.util.LuceneTokenizer

public class LuceneTokenizer
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	LuceneTokenizer.TokenizerType

Constructor Summary

Constructors

Constructor	Description
LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, boolean useStopFilter, LuceneAnalyzerUtil.StemFilterType stemFilterType)	Creates a tokenizer based on param values
LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, List<String> stopWords, boolean addToDefault, LuceneAnalyzerUtil.StemFilterType stemFilterType)	Creates a tokenizer based on param values
LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, LuceneAnalyzerUtil.StemFilterType stemFilterType, int mingram, int maxgram)	Creates a tokenizer for the ngram model based on param values

Method Summary

Modifier and Type	Method	Description
org.apache.lucene.analysis.TokenStream	getTokenStream()	get the tokenStream created by Tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

LuceneTokenizer

public LuceneTokenizer(String content,
 LuceneTokenizer.TokenizerType tokenizer,
 boolean useStopFilter,
 LuceneAnalyzerUtil.StemFilterType stemFilterType)

Creates a tokenizer based on param values

Parameters:

content - - The text to tokenize

tokenizer - - the type of tokenizer to use CLASSIC or DEFAULT

useStopFilter - - if set to true the token stream will be filtered using default Lucene stopset

stemFilterType - a preferred LuceneAnalyzerUtil.StemFilterType to use. Can be one of LuceneAnalyzerUtil.StemFilterType.PORTERSTEM_FILTER, LuceneAnalyzerUtil.StemFilterType.ENGLISHMINIMALSTEM_FILTER, or LuceneAnalyzerUtil.StemFilterType.NONE

LuceneTokenizer

public LuceneTokenizer(String content,
 LuceneTokenizer.TokenizerType tokenizer,
 List<String> stopWords,
 boolean addToDefault,
 LuceneAnalyzerUtil.StemFilterType stemFilterType)

Creates a tokenizer based on param values

Parameters:

content - - The text to tokenize

tokenizer - - the type of tokenizer to use CLASSIC or DEFAULT

stopWords - - Provide a set of user defined stop words

addToDefault - - If set to true, the stopSet words will be added to the Lucene default stop set. If false, then only the user provided words will be used as the stop set

stemFilterType - a preferred LuceneAnalyzerUtil.StemFilterType to use. Can be one of LuceneAnalyzerUtil.StemFilterType.PORTERSTEM_FILTER, LuceneAnalyzerUtil.StemFilterType.ENGLISHMINIMALSTEM_FILTER, or LuceneAnalyzerUtil.StemFilterType.NONE

LuceneTokenizer

public LuceneTokenizer(String content,
 LuceneTokenizer.TokenizerType tokenizer,
 LuceneAnalyzerUtil.StemFilterType stemFilterType,
 int mingram,
 int maxgram)

Creates a tokenizer for the ngram model based on param values

Parameters:

content - - The text to tokenize

tokenizer - - the type of tokenizer to use CLASSIC or DEFAULT

stemFilterType - - Type of stemming to perform

mingram - - Value of mingram for tokenizing

maxgram - - Value of maxgram for tokenizing

Method Details

getTokenStream

public org.apache.lucene.analysis.TokenStream getTokenStream()

get the tokenStream created by Tokenizer

Returns:

The TokenStream