Java.target.apidocs.org.antlr.v4.runtime.TokenSource.html Maven / Gradle / Ivy
Go to download
Show more of this group Show more artifacts with this name
Show all versions of antlr4-perf-testsuite Show documentation
Show all versions of antlr4-perf-testsuite Show documentation
The ANTLR 4 grammar compiler.
TokenSource (ANTLR 4 Runtime (Optimized) 4.13.1.3 API)
org.antlr.v4.runtime
Interface TokenSource
-
- All Known Implementing Classes:
- Lexer, LexerInterpreter, ListTokenSource, XPathLexer
public interface TokenSource
A source of tokens must provide a sequence of tokens via nextToken()
and also must reveal it's source of characters; CommonToken's text is
computed from a CharStream; it only store indices into the char
stream.
Errors from the lexer are never passed to the parser. Either you want to keep
going or you do not upon token recognition error. If you do not want to
continue lexing then you do not want to continue parsing. Just throw an
exception not under RecognitionException and Java will naturally toss
you all the way out of the recognizers. If you want to continue lexing then
you should not throw an exception to the parser--it has already requested a
token. Keep lexing until you get a valid one. Just report errors and keep
going, looking for a valid token.
-
-
Method Summary
All Methods Instance Methods Abstract Methods
Modifier and Type
Method and Description
intgetCharPositionInLine()
Get the index into the current line for the current position in the input
stream.
CharStreamgetInputStream()
Get the CharStream from which this token source is currently
providing tokens.
intgetLine()
Get the line number for the current position in the input stream.
StringgetSourceName()
Gets the name of the underlying input source.
TokenFactorygetTokenFactory()
Gets the TokenFactory this token source is currently using for
creating Token objects from the input.
TokennextToken()
Return a Token object from your input stream (usually a
CharStream).
voidsetTokenFactory(TokenFactory factory)
Set the TokenFactory this token source should use for creating
Token objects from the input.
-
-
Method Detail
-
nextToken
@NotNull
Token nextToken()
Return a Token object from your input stream (usually a
CharStream). Do not fail/return upon lexing error; keep chewing
on the characters until you get a good one; errors are not passed through
to the parser.
-
getLine
int getLine()
Get the line number for the current position in the input stream. The
first line in the input is line 1.
- Returns:
- The line number for the current position in the input stream, or
0 if the current token source does not track line numbers.
-
getCharPositionInLine
int getCharPositionInLine()
Get the index into the current line for the current position in the input
stream. The first character on a line has position 0.
- Returns:
- The line number for the current position in the input stream, or
-1 if the current token source does not track character positions.
-
getInputStream
@Nullable
CharStream getInputStream()
Get the CharStream from which this token source is currently
providing tokens.
- Returns:
- The
CharStream associated with the current position in
the input, or null if no input stream is available for the token
source.
-
getSourceName
@NotNull
String getSourceName()
Gets the name of the underlying input source. This method returns a
non-null, non-empty string. If such a name is not known, this method
returns IntStream.UNKNOWN_SOURCE_NAME.
-
setTokenFactory
void setTokenFactory(@NotNull
TokenFactory factory)
Set the TokenFactory this token source should use for creating
Token objects from the input.
- Parameters:
factory - The TokenFactory to use for creating tokens.
-
getTokenFactory
@NotNull
TokenFactory getTokenFactory()
Gets the TokenFactory this token source is currently using for
creating Token objects from the input.
- Returns:
- The
TokenFactory currently used by this token source.
Copyright © 1992–2023 Daniel Sun. All rights reserved.