cc.mallet.extract.Tokenization Maven / Gradle / Ivy

Go to download

Show more of this group Show more artifacts with this name
Show all versions of jcore-mallet-2.0.9 Show documentation

MALLET is a Java-based package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text.

The newest version!

/* Copyright (C) 2002 Univ. of Massachusetts Amherst, Computer Science Dept.
   This file is part of "MALLET" (MAchine Learning for LanguagE Toolkit).
   http://www.cs.umass.edu/~mccallum/mallet
   This software is provided under the terms of the Common Public License,
   version 1.0, as published by http://www.opensource.org.  For further
   information, see the file `LICENSE' included with this distribution. */

/** 
   @author Andrew McCallum [email protected]
 */

package cc.mallet.extract;

import cc.mallet.types.*;

public interface Tokenization extends Sequence //??
{
  /**
   * Returns the document of which this is a tokenization.
   */
	public Object getDocument ();

  public Span getSpan (int i);


  /** Returns a span formed by concatenating the spans from start to end.
   *  In more detail:
   *  
   *   The start of the new span will be the start index of getSpan(start).
   *   
The end of the new span will be the start index of getSpan(end).
   *   
Unless start == end, the new span will completely include getSpan(start).
   *   
The new span will never intersect getSpan(end)
   *   
If start == end, then the new span contains no text.
   *  
   *
   * @param start The index of the first token in the new span (inclusive).
   *   This is an index of a token, *not* an index into the document.
   * @param end The index of the first token in the new span (exclusive).
   *   This is an index of a token, *not* an index into the document.
   * @return A span into this tokenization's document
   */
  Span subspan (int start, int end);
}