Java.target.apidocs.org.antlr.v4.runtime.Lexer.html Maven / Gradle / Ivy
Go to download
Show more of this group Show more artifacts with this name
Show all versions of antlr4-perf-testsuite Show documentation
Show all versions of antlr4-perf-testsuite Show documentation
The ANTLR 4 grammar compiler.
Lexer (ANTLR 4 Runtime (Optimized) 4.7.3 API)
org.antlr.v4.runtime
Class Lexer
- java.lang.Object
-
- org.antlr.v4.runtime.Recognizer<Integer,LexerATNSimulator>
-
- org.antlr.v4.runtime.Lexer
-
- All Implemented Interfaces:
- TokenSource
- Direct Known Subclasses:
- LexerInterpreter, XPathLexer
public abstract class Lexer
extends Recognizer<Integer,LexerATNSimulator>
implements TokenSource
A lexer is recognizer that draws input symbols from a character stream.
lexer grammars result in a subclass of this object. A Lexer object
uses simplified match() and error recovery mechanisms in the interest
of speed.
-
-
Field Summary
Fields
Modifier and Type
Field and Description
int
_channel
The channel number for the current token
protected TokenFactory
_factory
How to create token objects
boolean
_hitEOF
Once we see EOF on char stream, next token will be EOF.
CharStream
_input
int
_mode
IntegerStack
_modeStack
String
_text
You can set the text for the current token to override what is in
the input char buffer.
Token
_token
The goal of all lexer rules/methods is to create a token object.
protected Tuple2<? extends TokenSource,CharStream>
_tokenFactorySourcePair
int
_tokenStartCharIndex
What character index in the stream did the current token start at?
Needed, for example, to get the text for current token.
int
_tokenStartCharPositionInLine
The character position of first character within the line
int
_tokenStartLine
The line on which the first character of the token resides
int
_type
The token type for the current token
static int
DEFAULT_MODE
static int
DEFAULT_TOKEN_CHANNEL
static int
HIDDEN
static int
MAX_CHAR_VALUE
static int
MIN_CHAR_VALUE
static int
MORE
static int
SKIP
-
Fields inherited from class org.antlr.v4.runtime.Recognizer
_interp, EOF
-
Constructor Summary
Constructors
Constructor and Description
Lexer(CharStream input)
-
Method Summary
Methods
Modifier and Type
Method and Description
Token
emit()
The standard method called to automatically emit a token at the
outermost lexical rule.
void
emit(Token token)
By default does not support multiple emits per nextToken invocation
for efficiency reasons.
Token
emitEOF()
List<? extends Token>
getAllTokens()
Return a list of all Token objects in input char stream.
int
getChannel()
String[]
getChannelNames()
String
getCharErrorDisplay(int c)
int
getCharIndex()
What is the index of the current character of lookahead?
int
getCharPositionInLine()
Get the index into the current line for the current position in the input
stream.
String
getErrorDisplay(int c)
String
getErrorDisplay(String s)
CharStream
getInputStream()
Get the CharStream
from which this token source is currently
providing tokens.
int
getLine()
Get the line number for the current position in the input stream.
String[]
getModeNames()
String
getSourceName()
Gets the name of the underlying input source.
String
getText()
Return the text matched so far for the current token or any
text override.
Token
getToken()
Override if emitting multiple tokens.
TokenFactory
getTokenFactory()
Gets the TokenFactory
this token source is currently using for
creating Token
objects from the input.
String[]
getTokenNames()
Deprecated.
int
getType()
void
mode(int m)
void
more()
Token
nextToken()
Return a token from this source; i.e., match a token on the char
stream.
void
notifyListeners(LexerNoViableAltException e)
int
popMode()
void
pushMode(int m)
void
recover(LexerNoViableAltException e)
void
recover(RecognitionException re)
Lexers can normally match any char in it's vocabulary after matching
a token, so do the easy thing and just kill a character and hope
it all works out.
void
reset()
void
setChannel(int channel)
void
setCharPositionInLine(int charPositionInLine)
void
setInputStream(CharStream input)
Set the char stream and reset the lexer
void
setLine(int line)
void
setText(String text)
Set the complete text of this token; it wipes any previous
changes to the text.
void
setToken(Token _token)
void
setTokenFactory(TokenFactory factory)
Set the TokenFactory
this token source should use for creating
Token
objects from the input.
void
setType(int ttype)
void
skip()
Instruct the lexer to skip creating a token for current lexer rule
and look for another token.
protected void
validateInputStream(ATN atn,
CharStream input)
-
Methods inherited from class org.antlr.v4.runtime.Recognizer
action, addErrorListener, getATN, getErrorHeader, getErrorListenerDispatch, getErrorListeners, getGrammarFileName, getInterpreter, getParseInfo, getRuleIndexMap, getRuleNames, getSerializedATN, getState, getTokenErrorDisplay, getTokenType, getTokenTypeMap, getVocabulary, precpred, removeErrorListener, removeErrorListeners, sempred, setInterpreter, setState
-
-
Field Detail
-
DEFAULT_MODE
public static final int DEFAULT_MODE
- See Also:
- Constant Field Values
-
MORE
public static final int MORE
- See Also:
- Constant Field Values
-
SKIP
public static final int SKIP
- See Also:
- Constant Field Values
-
DEFAULT_TOKEN_CHANNEL
public static final int DEFAULT_TOKEN_CHANNEL
- See Also:
- Constant Field Values
-
HIDDEN
public static final int HIDDEN
- See Also:
- Constant Field Values
-
MIN_CHAR_VALUE
public static final int MIN_CHAR_VALUE
- See Also:
- Constant Field Values
-
MAX_CHAR_VALUE
public static final int MAX_CHAR_VALUE
- See Also:
- Constant Field Values
-
_input
public CharStream _input
-
_tokenFactorySourcePair
protected Tuple2<? extends TokenSource,CharStream> _tokenFactorySourcePair
-
_factory
protected TokenFactory _factory
How to create token objects
-
_token
public Token _token
The goal of all lexer rules/methods is to create a token object.
This is an instance variable as multiple rules may collaborate to
create a single token. nextToken will return this object after
matching lexer rule(s). If you subclass to allow multiple token
emissions, then set this to the last token to be matched or
something nonnull so that the auto token emit mechanism will not
emit another token.
-
_tokenStartCharIndex
public int _tokenStartCharIndex
What character index in the stream did the current token start at?
Needed, for example, to get the text for current token. Set at
the start of nextToken.
-
_tokenStartLine
public int _tokenStartLine
The line on which the first character of the token resides
-
_tokenStartCharPositionInLine
public int _tokenStartCharPositionInLine
The character position of first character within the line
-
_hitEOF
public boolean _hitEOF
Once we see EOF on char stream, next token will be EOF.
If you have DONE : EOF ; then you see DONE EOF.
-
_channel
public int _channel
The channel number for the current token
-
_type
public int _type
The token type for the current token
-
_modeStack
public final IntegerStack _modeStack
-
_mode
public int _mode
-
_text
public String _text
You can set the text for the current token to override what is in
the input char buffer. Use setText() or can set this instance var.
-
Constructor Detail
-
Lexer
public Lexer(CharStream input)
-
Method Detail
-
reset
public void reset()
-
nextToken
public Token nextToken()
Return a token from this source; i.e., match a token on the char
stream.
- Specified by:
nextToken
in interface TokenSource
-
skip
public void skip()
Instruct the lexer to skip creating a token for current lexer rule
and look for another token. nextToken() knows to keep looking when
a lexer rule finishes with token set to SKIP_TOKEN. Recall that
if token==null at end of any token rule, it creates one for you
and emits it.
-
more
public void more()
-
mode
public void mode(int m)
-
pushMode
public void pushMode(int m)
-
popMode
public int popMode()
-
getTokenFactory
public TokenFactory getTokenFactory()
Description copied from interface: TokenSource
Gets the TokenFactory
this token source is currently using for
creating Token
objects from the input.
- Specified by:
getTokenFactory
in interface TokenSource
- Returns:
- The
TokenFactory
currently used by this token source.
-
setTokenFactory
public void setTokenFactory(TokenFactory factory)
Description copied from interface: TokenSource
Set the TokenFactory
this token source should use for creating
Token
objects from the input.
- Specified by:
setTokenFactory
in interface TokenSource
- Parameters:
factory
- The TokenFactory
to use for creating tokens.
-
setInputStream
public void setInputStream(CharStream input)
Set the char stream and reset the lexer
-
validateInputStream
protected void validateInputStream(ATN atn,
CharStream input)
-
getSourceName
public String getSourceName()
Description copied from interface: TokenSource
Gets the name of the underlying input source. This method returns a
non-null, non-empty string. If such a name is not known, this method
returns IntStream.UNKNOWN_SOURCE_NAME
.
- Specified by:
getSourceName
in interface TokenSource
-
getInputStream
public CharStream getInputStream()
Description copied from interface: TokenSource
Get the CharStream
from which this token source is currently
providing tokens.
- Specified by:
getInputStream
in interface TokenSource
- Specified by:
getInputStream
in class Recognizer<Integer,LexerATNSimulator>
- Returns:
- The
CharStream
associated with the current position in
the input, or null
if no input stream is available for the token
source.
-
emit
public void emit(Token token)
By default does not support multiple emits per nextToken invocation
for efficiency reasons. Subclass and override this method, nextToken,
and getToken (to push tokens into a list and pull from that list
rather than a single variable as this implementation does).
-
emit
public Token emit()
The standard method called to automatically emit a token at the
outermost lexical rule. The token object should point into the
char buffer start..stop. If there is a text override in 'text',
use that to set the token's text. Override this method to emit
custom Token objects or provide a new factory.
-
emitEOF
public Token emitEOF()
-
getLine
public int getLine()
Description copied from interface: TokenSource
Get the line number for the current position in the input stream. The
first line in the input is line 1.
- Specified by:
getLine
in interface TokenSource
- Returns:
- The line number for the current position in the input stream, or
0 if the current token source does not track line numbers.
-
getCharPositionInLine
public int getCharPositionInLine()
Description copied from interface: TokenSource
Get the index into the current line for the current position in the input
stream. The first character on a line has position 0.
- Specified by:
getCharPositionInLine
in interface TokenSource
- Returns:
- The line number for the current position in the input stream, or
-1 if the current token source does not track character positions.
-
setLine
public void setLine(int line)
-
setCharPositionInLine
public void setCharPositionInLine(int charPositionInLine)
-
getCharIndex
public int getCharIndex()
What is the index of the current character of lookahead?
-
getText
public String getText()
Return the text matched so far for the current token or any
text override.
-
setText
public void setText(String text)
Set the complete text of this token; it wipes any previous
changes to the text.
-
getToken
public Token getToken()
Override if emitting multiple tokens.
-
setToken
public void setToken(Token _token)
-
setType
public void setType(int ttype)
-
getType
public int getType()
-
setChannel
public void setChannel(int channel)
-
getChannel
public int getChannel()
-
getChannelNames
@Nullable
public String[] getChannelNames()
-
getModeNames
@Nullable
public String[] getModeNames()
-
getTokenNames
@Deprecated
public String[] getTokenNames()
Deprecated.
Used to print out token names like ID during debugging and
error reporting. The generated parsers implement a method
that overrides this to point to their String[] tokenNames.
- Specified by:
getTokenNames
in class Recognizer<Integer,LexerATNSimulator>
-
getAllTokens
public List<? extends Token> getAllTokens()
Return a list of all Token objects in input char stream.
Forces load of all tokens. Does not include EOF token.
-
recover
public void recover(LexerNoViableAltException e)
-
notifyListeners
public void notifyListeners(LexerNoViableAltException e)
-
getErrorDisplay
public String getErrorDisplay(int c)
-
getCharErrorDisplay
public String getCharErrorDisplay(int c)
-
recover
public void recover(RecognitionException re)
Lexers can normally match any char in it's vocabulary after matching
a token, so do the easy thing and just kill a character and hope
it all works out. You can instead use the rule invocation stack
to do sophisticated error recovery if you are in a fragment rule.
Copyright © 1992–2019 Tunnel Vision Laboratories, LLC. All rights reserved.