net.sf.eBus.text.package-info Maven / Gradle / Ivy
/*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later
* version.
*
* This library is distributed in the hope that it will be
* useful, but WITHOUT ANY WARRANTY; without even the implied
* warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
* PURPOSE. See the GNU Lesser General Public License for more
* details.
*
* You should have received a copy of the GNU Lesser General
* Public License along with this library; if not, write to the
*
* Free Software Foundation, Inc.,
* 59 Temple Place, Suite 330,
* Boston, MA
* 02111-1307 USA
*
* The Initial Developer of the Original Code is Charles W. Rapp.
* Portions created by Charles W. Rapp are
* Copyright 2012. Charles W. Rapp
* All Rights Reserved.
*/
/**
* {@link net.sf.eBus.text.TokenLexer} takes a given input and
* returns a series of analyzed tokens. These tokens contain:
*
* -
* A token type. May be either a pre-defined or
* user-defined.
*
* -
* The token text value as collected from the input.
*
* -
* The Java object created from the token text.
*
* -
* The line number on which the token appears.
*
*
*
* The most important difference between {@code Lexer} and
* {@code java.util.StringTokenizer} is that Lexer analyzes the
* tokens into types. Further, tokens are not created by
* breaking a line based on inter-field separator characters or
* by regular expression. Instead, Lexer searches for tokens
* based on types. This lexical package is designed to work with
* a higher level parser rather than a line-by-line text
* processor.
*/
package net.sf.eBus.text;