• Home
• org.apache.lucene
• lucene-backward-codecs
• 10.1.0
• source code
• package-info.java

×

Project download

Many resources are needed to download a project. Please understand that we have to compensate our server costs. Thank you in advance.
Project price only 1 $

You can buy this project and download/modify it how often you want.

org.apache.lucene.backward_codecs.lucene99.package-info Maven / Gradle / Ivy

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

/**
 * Lucene 9.9 file format.
 *
 * 
Apache Lucene - Index File Formats
 *
 * 

 *
 * 

 *   
Introduction
 *   
Definitions
 *       

 *         
Inverted Indexing
 *         
Types of Fields
 *         
Segments
 *         
Document Numbers
 *       
 *   
Index Structure Overview
 *   
File Naming
 *   
Summary of File Extensions
 *       

 *         
Lock File
 *         
History
 *         
Limitations
 *       
 * 
 *
 * 
 
 *
 * 
Introduction
 *
 * 

 *
 * 
This document defines the index file formats used in this version of Lucene. If you are using
 * a different version of Lucene, please consult the copy of docs/ that was distributed
 * with the version you are using.
 *
 * 
This document attempts to provide a high-level definition of the Apache Lucene file formats.
 * 
 
 *
 * 
Definitions
 *
 * 

 *
 * 
The fundamental concepts in Lucene are index, document, field and term.
 *
 * 
An index contains a sequence of documents.
 *
 * 

 *   
A document is a sequence of fields.
 *   
A field is a named sequence of terms.
 *   
A term is a sequence of bytes.
 * 
 *
 * 
The same sequence of bytes in two different fields is considered a different term. Thus terms
 * are represented as a pair: the string naming the field, and the bytes within the field. 
 *
 * 
Inverted Indexing
 *
 * 
Lucene's index stores terms and statistics about those terms in order to make term-based
 * search more efficient. Lucene's terms index falls into the family of indexes known as an
 * inverted index. This is because it can list, for a term, the documents that contain it.
 * This is the inverse of the natural relationship, in which documents list terms. 
 *
 * 
Types of Fields
 *
 * 
In Lucene, fields may be stored, in which case their text is stored in the index
 * literally, in a non-inverted manner. Fields that are inverted are called indexed. A field
 * may be both stored and indexed.
 *
 * 
The text of a field may be tokenized into terms to be indexed, or the text of a field
 * may be used literally as a term to be indexed. Most fields are tokenized, but sometimes it is
 * useful for certain identifier fields to be indexed literally.
 *
 * 
See the {@link org.apache.lucene.document.Field Field} java docs for more information on
 * Fields. 
 *
 * 
Segments
 *
 * 
Lucene indexes may be composed of multiple sub-indexes, or segments. Each segment is a
 * fully independent index, which could be searched separately. Indexes evolve by:
 *
 * 

 *   
Creating new segments for newly added documents.
 *   
Merging existing segments.
 * 
 *
 * 
Searches may involve multiple segments and/or multiple indexes, each index potentially
 * composed of a set of segments. 
 *
 * 
Document Numbers
 *
 * 
Internally, Lucene refers to documents by an integer document number. The first
 * document added to an index is numbered zero, and each subsequent document added gets a number one
 * greater than the previous.
 *
 * 
Note that a document's number may change, so caution should be taken when storing these
 * numbers outside of Lucene. In particular, numbers may change in the following situations:
 *
 * 

 *   

 *       
The numbers stored in each segment are unique only within the segment, and must be
 *       converted before they can be used in a larger context. The standard technique is to
 *       allocate each segment a range of values, based on the range of numbers used in that
 *       segment. To convert a document number from a segment to an external value, the segment's
 *       base document number is added. To convert an external value back to a
 *       segment-specific value, the segment is identified by the range that the external value is
 *       in, and the segment's base value is subtracted. For example two five document segments
 *       might be combined, so that the first segment has a base value of zero, and the second of
 *       five. Document three from the second segment would have an external value of eight.
 *   

 *       
When documents are deleted, gaps are created in the numbering. These are eventually
 *       removed as the index evolves through merging. Deleted documents are dropped when segments
 *       are merged. A freshly-merged segment thus has no gaps in its numbering.
 * 
 *
 * 
 
 *
 * 
Index Structure Overview
 *
 * 

 *
 * 
Each segment index maintains the following:
 *
 * 

 *   
{@link org.apache.lucene.codecs.lucene99.Lucene99SegmentInfoFormat Segment info}. This
 *       contains metadata about a segment, such as the number of documents, what files it uses, and
 *       information about how the segment is sorted
 *   
{@link org.apache.lucene.codecs.lucene94.Lucene94FieldInfosFormat Field names}. This
 *       contains metadata about the set of named fields used in the index.
 *   
{@link org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat Stored Field values}.
 *       This contains, for each document, a list of attribute-value pairs, where the attributes are
 *       field names. These are used to store auxiliary information about the document, such as its
 *       title, url, or an identifier to access a database. The set of stored fields are what is
 *       returned for each hit when searching. This is keyed by document number.
 *   
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Term dictionary}.
 *       A dictionary containing all of the terms used in all of the indexed fields of all of the
 *       documents. The dictionary also contains the number of documents which contain the term, and
 *       pointers to the term's frequency and proximity data.
 *   
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Term Frequency
 *       data}. For each term in the dictionary, the numbers of all the documents that contain that
 *       term, and the frequency of the term in that document, unless frequencies are omitted
 *       ({@link org.apache.lucene.index.IndexOptions#DOCS IndexOptions.DOCS})
 *   
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Term Proximity
 *       data}. For each term in the dictionary, the positions that the term occurs in each
 *       document. Note that this will not exist if all fields in all documents omit position data.
 *   
{@link org.apache.lucene.codecs.lucene90.Lucene90NormsFormat Normalization factors}. For
 *       each field in each document, a value is stored that is multiplied into the score for hits
 *       on that field.
 *   
{@link org.apache.lucene.codecs.lucene90.Lucene90TermVectorsFormat Term Vectors}. For each
 *       field in each document, the term vector (sometimes called document vector) may be stored. A
 *       term vector consists of term text and term frequency. To add Term Vectors to your index see
 *       the {@link org.apache.lucene.document.Field Field} constructors
 *   
{@link org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat Per-document values}. Like
 *       stored values, these are also keyed by document number, but are generally intended to be
 *       loaded into main memory for fast access. Whereas stored values are generally intended for
 *       summary results from searches, per-document values are useful for things like scoring
 *       factors.
 *   
{@link org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat Live documents}. An
 *       optional file indicating which documents are live.
 *   
{@link org.apache.lucene.codecs.lucene90.Lucene90PointsFormat Point values}. Optional pair
 *       of files, recording dimensionally indexed fields, to enable fast numeric range filtering
 *       and large numeric values like BigInteger and BigDecimal (1D) and geographic shape
 *       intersection (2D, 3D).
 *   
{@link org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat Vector values}. The
 *       vector format stores numeric vectors in a format optimized for random access and
 *       computation, supporting high-dimensional nearest-neighbor search.
 * 
 *
 * 
Details on each of these are provided in their linked pages. 
 
 *
 * 
File Naming
 *
 * 

 *
 * 
All files belonging to a segment have the same name with varying extensions. The extensions
 * correspond to the different file formats described below. When using the Compound File format
 * (default for small segments) these files (except for the Segment info file, the Lock file, and
 * Deleted documents file) are collapsed into a single .cfs file (see below for details)
 *
 * 
Typically, all segments in an index are stored in a single directory, although this is not
 * required.
 *
 * 
File names are never re-used. That is, when any file is saved to the Directory it is given a
 * never before used filename. This is achieved using a simple generations approach. For example,
 * the first segments file is segments_1, then segments_2, etc. The generation is a sequential long
 * integer represented in alpha-numeric (base 36) form. 
 
 *
 * 
Summary of File Extensions
 *
 * 

 *
 * 
The following table summarizes the names and extensions of the files in Lucene:
 *
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
lucene filenames by extension
Name
Extension
Brief Description
{@link org.apache.lucene.index.SegmentInfos Segments File}
segments_N
Stores information about a commit point
Lock File
write.lock
The Write lock prevents multiple IndexWriters from writing to the same
 * file.
{@link org.apache.lucene.codecs.lucene99.Lucene99SegmentInfoFormat Segment Info}
.si
Stores metadata about a segment
{@link org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat Compound File}
.cfs, .cfe
An optional "virtual" file consisting of all the other index files for
 * systems that frequently run out of file handles.
{@link org.apache.lucene.codecs.lucene94.Lucene94FieldInfosFormat Fields}
.fnm
Stores information about the fields
{@link org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat Field Index}
.fdx
Contains pointers to field data
{@link org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat Field Data}
.fdt
The stored fields for documents
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Term Dictionary}
.tim
The term dictionary, stores term info
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Term Index}
.tip
The index into the Term Dictionary
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Frequencies}
.doc
Contains the list of docs which contain each term along with frequency
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Positions}
.pos
Stores position information about where a term occurs in the index
{@link org.apache.lucene.backward_codecs.lucene99.Lucene99PostingsFormat Payloads}
.pay
Stores additional per-position metadata information such as character offsets and user payloads
{@link org.apache.lucene.codecs.lucene90.Lucene90NormsFormat Norms}
.nvd, .nvm
Encodes length and boost factors for docs and fields
{@link org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat Per-Document Values}
.dvd, .dvm
Encodes additional scoring factors or other per-document information.
{@link org.apache.lucene.codecs.lucene90.Lucene90TermVectorsFormat Term Vector Index}
.tvx
Stores offset into the document data file
{@link org.apache.lucene.codecs.lucene90.Lucene90TermVectorsFormat Term Vector Data}
.tvd
Contains term vector data.
{@link org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat Live Documents}
.liv
Info about what documents are live
{@link org.apache.lucene.codecs.lucene90.Lucene90PointsFormat Point values}
.dii, .dim
Holds indexed points
{@link org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat Vector values}
.vec, .vem, .veq, vex
Holds indexed vectors; .vec files contain the raw vector data,
 * .vem the vector metadata, .veq the quantized vector data, and .vex the
 * hnsw graph data.
 *
 * 
 
 *
 * 
Lock File
 *
 * The write lock, which is stored in the index directory by default, is named "write.lock". If the
 * lock directory is different from the index directory then the write lock will be named
 * "XXXX-write.lock" where XXXX is a unique prefix derived from the full path to the index
 * directory. When this file is present, a writer is currently modifying the index (adding or
 * removing documents). This lock file ensures that only one writer is modifying the index at a
 * time. 
 *
 * 
History
 *
 * 
Compatibility notes are provided in this document, describing how file formats have changed
 * from prior versions:
 *
 * 

 *   
In version 2.1, the file format was changed to allow lock-less commits (ie, no more commit
 *       lock). The change is fully backwards compatible: you can open a pre-2.1 index for searching
 *       or adding/deleting of docs. When the new segments file is saved (committed), it will be
 *       written in the new file format (meaning no specific "upgrade" process is needed). But note
 *       that once a commit has occurred, pre-2.1 Lucene will not be able to read the index.
 *   
In version 2.3, the file format was changed to allow segments to share a single set of doc
 *       store (vectors & stored fields) files. This allows for faster indexing in certain
 *       cases. The change is fully backwards compatible (in the same way as the lock-less commits
 *       change in 2.1).
 *   
In version 2.4, Strings are now written as true UTF-8 byte sequence, not Java's modified
 *       UTF-8. See LUCENE-510 for
 *       details.
 *   
In version 2.9, an optional opaque Map<String,String> CommitUserData may be passed to
 *       IndexWriter's commit methods (and later retrieved), which is recorded in the segments_N
 *       file. See LUCENE-1382 for
 *       details. Also, diagnostics were added to each segment written recording details about why
 *       it was written (due to flush, merge; which OS/JRE was used; etc.). See issue LUCENE-1654 for details.
 *   
In version 3.0, compressed fields are no longer written to the index (they can still be
 *       read, but on merge the new segment will write them, uncompressed). See issue LUCENE-1960 for details.
 *   
In version 3.1, segments records the code version that created them. See LUCENE-2720 for details.
 *       Additionally segments track explicitly whether or not they have term vectors. See LUCENE-2811 for details.
 *   
In version 3.2, numeric fields are written as natively to stored fields file, previously
 *       they were stored in text format only.
 *   
In version 3.4, fields can omit position data while still indexing term frequencies.
 *   
In version 4.0, the format of the inverted index became extensible via the {@link
 *       org.apache.lucene.codecs.Codec Codec} api. Fast per-document storage ({@code DocValues})
 *       was introduced. Normalization factors need no longer be a single byte, they can be any
 *       {@link org.apache.lucene.index.NumericDocValues NumericDocValues}. Terms need not be
 *       unicode strings, they can be any byte sequence. Term offsets can optionally be indexed into
 *       the postings lists. Payloads can be stored in the term vectors.
 *   
In version 4.1, the format of the postings list changed to use either of FOR compression or
 *       variable-byte encoding, depending upon the frequency of the term. Terms appearing only once
 *       were changed to inline directly into the term dictionary. Stored fields are compressed by
 *       default.
 *   
In version 4.2, term vectors are compressed by default. DocValues has a new multi-valued
 *       type (SortedSet), that can be used for faceting/grouping/joining on multi-valued fields.
 *   
In version 4.5, DocValues were extended to explicitly represent missing values.
 *   
In version 4.6, FieldInfos were extended to support per-field DocValues generation, to
 *       allow updating NumericDocValues fields.
 *   
In version 4.8, checksum footers were added to the end of each index file for improved data
 *       integrity. Specifically, the last 8 bytes of every index file contain the zlib-crc32
 *       checksum of the file.
 *   
In version 4.9, DocValues has a new multi-valued numeric type (SortedNumeric) that is
 *       suitable for faceting/sorting/analytics.
 *   
In version 5.4, DocValues have been improved to store more information on disk: addresses
 *       for binary fields and ord indexes for multi-valued fields.
 *   
In version 6.0, Points were added, for multi-dimensional range/distance search.
 *   
In version 6.2, new Segment info format that reads/writes the index sort, to support index
 *       sorting.
 *   
In version 7.0, DocValues have been improved to better support sparse doc values thanks to
 *       an iterator API.
 *   
In version 8.0, postings have been enhanced to record, for each block of doc ids, the (term
 *       freq, normalization factor) pairs that may trigger the maximum score of the block. This
 *       information is recorded alongside skip data in order to be able to skip blocks of doc ids
 *       if they may not produce high enough scores. Additionally doc values and norms has been
 *       extended with jump-tables to make access O(1) instead of O(n), where n is the number of
 *       elements to skip when advancing in the data.
 *   
In version 8.4, postings, positions, offsets and payload lengths have move to a more
 *       performant encoding that is vectorized.
 *   
In version 8.6, index sort serialization is delegated to the sorts themselves, to allow
 *       user-defined sorts to be used
 *   
In version 8.7, stored fields compression became adaptive to better handle documents with
 *       smaller stored fields.
 *   
In version 9.0, vector-valued fields were added.
 *   
In version 9.1, vector-valued fields were modified to add a graph hierarchy.
 *   
In version 9.2, docs of vector-valued fields were moved from .vem to .vec and encoded by
 *       IndexDISI. ordToDoc mappings was added to .vem.
 *   
In version 9.5, HNSW graph connections were changed to be delta-encoded with vints.
 *       Additionally, metadata file size improvements were made by delta-encoding nodes by graph
 *       layer and not writing the node ids for the zeroth layer.
 *   
In version 9.9, Vector scalar quantization support was added. Allowing the HNSW vector
 *       format to utilize int8 quantized vectors for float32 vector search.
 * 
 *
 * 
 *
 * 
Limitations
 *
 * 

 *
 * 
Lucene uses a Java int to refer to document numbers, and the index file format
 * uses an Int32 on-disk to store document numbers. This is a limitation of both the
 * index file format and the current implementation. Eventually these should be replaced with either
 * UInt64 values, or better yet, {@link org.apache.lucene.store.DataOutput#writeVInt
 * VInt} values which have no limit. 
 */
package org.apache.lucene.backward_codecs.lucene99;