Uses of Interface
opennlp.tools.tokenize.Tokenizer
-
Packages that use Tokenizer Package Description opennlp.tools.cmdline.parser opennlp.tools.formats.brat Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).opennlp.tools.formats.muc Experimental package related to theMUC
corpus format.opennlp.tools.tokenize Contains classes related to finding token or words in a string.opennlp.tools.util.featuregen This package contains classes for generating sequence features. -
-
Uses of Tokenizer in opennlp.tools.cmdline.parser
Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer Modifier and Type Method Description static Parse[]
ParserTool. parseLine(String line, Parser parser, Tokenizer tokenizer, int numParses)
-
Uses of Tokenizer in opennlp.tools.formats.brat
Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer Constructor Description BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer)
BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)
BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)
Creates a newBratNameSampleStream
.BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)
Creates a newBratNameSampleStream
. -
Uses of Tokenizer in opennlp.tools.formats.muc
Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer Constructor Description MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)
Initializes aMucNameContentHandler
.MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)
Initializes aMucNameSampleStream
. -
Uses of Tokenizer in opennlp.tools.tokenize
Classes in opennlp.tools.tokenize that implement Tokenizer Modifier and Type Class Description class
SimpleTokenizer
A basicTokenizer
implementation which performs tokenization using character classes.class
TokenizerME
ATokenizer
for converting raw text into separated tokens.class
WhitespaceTokenizer
A basicTokenizer
implementation which performs tokenization using white spaces.class
WordpieceTokenizer
ATokenizer
implementation which performs tokenization using word pieces.Constructors in opennlp.tools.tokenize with parameters of type Tokenizer Constructor Description TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)
Initializes an instance to evaluate aTokenizer
.TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)
Initializes ainstance
. -
Uses of Tokenizer in opennlp.tools.util.featuregen
Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer Constructor Description TokenPatternFeatureGenerator(Tokenizer supportTokenizer)
Initializes aTokenPatternFeatureGenerator
instance.
-