org.modelcc.lexer.Tokenizer Maven / Gradle / Ivy

Go to download

Show more of this group Show more artifacts with this name
Show all versions of ModelCC Show documentation

ModelCC is a model-based parser generator (a.k.a. compiler compiler) that decouples language specification from language processing, avoiding some of the problems caused by grammar-driven parser generators. ModelCC receives a conceptual model as input, along with constraints that annotate it. It is then able to create a parser for the desired textual language and the generated parser fully automates the instantiation of the language conceptual model. ModelCC also includes a built-in reference resolution mechanism that results in abstract syntax graphs, rather than mere abstract syntax trees.

The newest version!

/*
 * ModelCC, distributed under ModelCC Shared Software License, www.modelcc.org
 */

package org.modelcc.lexer;

import java.io.Reader;

import org.modelcc.language.lexis.LexicalSpecification;

/**
 * ModelCC tokenizer abstract class.
 * 
 * @author Fernando Berzal ([email protected])
 */
public abstract class Tokenizer 
{
    /**
     * Lexical specification
     */
	protected LexicalSpecification lexis;
	
	/** 
	 * Input reader
	 */
	protected Reader reader; 
	
	/**
	 * Constructor
	 * @param lexis Language lexical specification
	 * @param reader Tokenizer input reader
	 */
	protected Tokenizer (LexicalSpecification lexis, Reader reader)
	{
		this.lexis = lexis;
		this.reader = reader;
	}
		
    /**
     * Scans input to obtain a stream of tokens.
     * @return the next token from this tokenizeer.
     */
    public abstract Token nextToken();
}