indorm.lucene-analyzers-opennlp.8.10.2.source-code.overview.html Maven / Gradle / Ivy
Apache Lucene OpenNLP integration module
This module exposes functionality from
Apache OpenNLP to Apache Lucene.
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text.
For an introduction to Lucene's analysis API, see the {@link org.apache.lucene.analysis} package documentation.
The OpenNLP Tokenizer behavior is similar to the WhiteSpaceTokenizer but is smart about
inter-word punctuation. The term stream looks very much like the way you parse words and
punctuation while reading. The major difference between this tokenizer and most other
tokenizers shipped with Lucene is that punctuation is tokenized. This is required for
the following taggers to operate properly.
The OpenNLP taggers annotate terms using the TypeAttribute.
OpenNLPTokenizer segments text into sentences or words. This Tokenizer
uses the OpenNLP Sentence Detector and/or Tokenizer classes. When used together, the
Tokenizer receives sentences and can do a better job.OpenNLPFilter tags words using one or more technologies: Part-of-Speech,
Chunking, and Named Entity Recognition. These tags are assigned as token types. Note that
only of these operations will tag
Since the TypeAttribute is not stored in the index, it is recommended that one
of these filters is used following OpenNLPFilter to enable search against the
assigned tags:
TypeAsPayloadFilter copies the TypeAttribute value to the
PayloadAttributeTypeAsSynonymFilter creates a cloned token at the same position as each
tagged token, and copies the {{TypeAttribute}} value to the {{CharTermAttribute}}, optionally
with a customized prefix (so that tags effectively occupy a different namespace from token
text).