Uses of Class
opennlp.tools.tokenize.TokenSample
-
Packages that use TokenSample Package Description opennlp.tools.cmdline.tokenizer opennlp.tools.formats Experimental package related to converting various corpora to OpenNLP Format.opennlp.tools.formats.ad Experimental package related to theArvores Deitadas corpus
format.opennlp.tools.formats.conllu Experimental package related to the CoNNL-U format.opennlp.tools.formats.convert opennlp.tools.formats.irishsentencebank Experimental package related to the Irish Sentence Bank format.opennlp.tools.formats.masc Experimental package related to theMASC
corpus format.opennlp.tools.tokenize Contains classes related to finding token or words in a string.opennlp.tools.tokenize.lang.en -
-
Uses of TokenSample in opennlp.tools.cmdline.tokenizer
Methods in opennlp.tools.cmdline.tokenizer with parameters of type TokenSample Modifier and Type Method Description void
DetokenEvaluationErrorListener. misclassified(TokenSample reference, TokenSample prediction)
void
TokenEvaluationErrorListener. misclassified(TokenSample reference, TokenSample prediction)
-
Uses of TokenSample in opennlp.tools.formats
Methods in opennlp.tools.formats that return types with arguments of type TokenSample Modifier and Type Method Description ObjectStream<TokenSample>
ConllXTokenSampleStreamFactory. create(String[] args)
ObjectStream<TokenSample>
TokenSampleStreamFactory. create(String[] args)
-
Uses of TokenSample in opennlp.tools.formats.ad
Methods in opennlp.tools.formats.ad that return types with arguments of type TokenSample Modifier and Type Method Description ObjectStream<TokenSample>
ADTokenSampleStreamFactory. create(String[] args)
-
Uses of TokenSample in opennlp.tools.formats.conllu
Methods in opennlp.tools.formats.conllu that return TokenSample Modifier and Type Method Description TokenSample
ConlluTokenSampleStream. read()
Methods in opennlp.tools.formats.conllu that return types with arguments of type TokenSample Modifier and Type Method Description ObjectStream<TokenSample>
ConlluTokenSampleStreamFactory. create(String[] args)
-
Uses of TokenSample in opennlp.tools.formats.convert
Methods in opennlp.tools.formats.convert that return TokenSample Modifier and Type Method Description TokenSample
NameToTokenSampleStream. read()
TokenSample
POSToTokenSampleStream. read()
Methods in opennlp.tools.formats.convert that return types with arguments of type TokenSample Modifier and Type Method Description ObjectStream<TokenSample>
NameToTokenSampleStreamFactory. create(String[] args)
ObjectStream<TokenSample>
ParseToTokenSampleStreamFactory. create(String[] args)
ObjectStream<TokenSample>
POSToTokenSampleStreamFactory. create(String[] args)
-
Uses of TokenSample in opennlp.tools.formats.irishsentencebank
Methods in opennlp.tools.formats.irishsentencebank that return TokenSample Modifier and Type Method Description TokenSample
IrishSentenceBankDocument.IrishSentenceBankSentence. getTokenSample()
Methods in opennlp.tools.formats.irishsentencebank that return types with arguments of type TokenSample Modifier and Type Method Description ObjectStream<TokenSample>
IrishSentenceBankTokenSampleStreamFactory. create(String[] args)
-
Uses of TokenSample in opennlp.tools.formats.masc
Methods in opennlp.tools.formats.masc that return TokenSample Modifier and Type Method Description TokenSample
MascTokenSampleStream. read()
Methods in opennlp.tools.formats.masc that return types with arguments of type TokenSample Modifier and Type Method Description ObjectStream<TokenSample>
MascTokenSampleStreamFactory. create(String[] args)
-
Uses of TokenSample in opennlp.tools.tokenize
Methods in opennlp.tools.tokenize that return TokenSample Modifier and Type Method Description static TokenSample
TokenSample. parse(String sampleString, String separatorChars)
Parses a string sample.protected TokenSample
DetokenizerEvaluator. processSample(TokenSample reference)
protected TokenSample
TokenizerEvaluator. processSample(TokenSample reference)
TokenSample
TokenizerStream. read()
TokenSample
TokenSampleStream. read()
Methods in opennlp.tools.tokenize with parameters of type TokenSample Modifier and Type Method Description protected Iterator<Event>
TokSpanEventStream. createEvents(TokenSample tokenSample)
Adds training events to the event stream for each of the specifiedsample
.protected TokenSample
DetokenizerEvaluator. processSample(TokenSample reference)
protected TokenSample
TokenizerEvaluator. processSample(TokenSample reference)
Method parameters in opennlp.tools.tokenize with type arguments of type TokenSample Modifier and Type Method Description void
TokenizerCrossValidator. evaluate(ObjectStream<TokenSample> samples, int nFolds)
Starts the evaluation.static TokenizerModel
TokenizerME. train(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams)
Trains a model for theTokenizerME
.Constructor parameters in opennlp.tools.tokenize with type arguments of type TokenSample Constructor Description TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics)
Initializes a new event stream based on the data stream using aTokenContextGenerator
that relies on aDefaultTokenContextGenerator
.TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, TokenContextGenerator cg)
Initializes a new event stream based on the data stream using aTokenContextGenerator
.TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg)
Initializes a new event stream based on the data stream using aTokenContextGenerator
.WhitespaceTokenStream(ObjectStream<TokenSample> tokens)
Initializes aWhitespaceTokenStream
. -
Uses of TokenSample in opennlp.tools.tokenize.lang.en
Methods in opennlp.tools.tokenize.lang.en that return TokenSample Modifier and Type Method Description TokenSample
TokenSampleStream. next()
-