Java.target.apidocs.org.antlr.v4.runtime.BufferedTokenStream.html Maven / Gradle / Ivy

Go to download

Show more of this group Show more artifacts with this name
Show all versions of antlr4-perf-testsuite Show documentation

The ANTLR 4 grammar compiler.

There is a newer version: 4.9.0

Show newest version







BufferedTokenStream (ANTLR 4 Runtime (Optimized) 4.7.3 API)













Overview
Package
Class
Use
Tree
Deprecated
Index
Help




Prev Class
Next Class


Frames
No Frames


All Classes






Summary: 
Nested | 
Field | 
Constr | 
Method


Detail: 
Field | 
Constr | 
Method








org.antlr.v4.runtime
Class BufferedTokenStream



java.lang.Object


org.antlr.v4.runtime.BufferedTokenStream







All Implemented Interfaces:
IntStream, TokenStream


Direct Known Subclasses:
CommonTokenStream




public class BufferedTokenStream
extends Object
implements TokenStream
This implementation of TokenStream loads tokens from a
 TokenSource on-demand, and places the tokens in a buffer to provide
 access to any previous token by index.

 
 This token stream ignores the value of Token.getChannel(). If your
 parser requires the token stream filter tokens to only those on a particular
 channel, such as Token.DEFAULT_CHANNEL or
 Token.HIDDEN_CHANNEL, use a filtering token stream such a
 CommonTokenStream.











Field Summary

Fields 

Modifier and Type
Field and Description


protected boolean
fetchedEOF
Indicates whether the Token.EOF token has been fetched from
 tokenSource and added to tokens.



protected int
p
The index into tokens of the current token (next token to
 consume()).



protected List<Token>
tokens
A collection of all tokens fetched from the token source.



protected TokenSource
tokenSource
The TokenSource from which tokens for this stream are fetched.







Fields inherited from interface org.antlr.v4.runtime.IntStream
EOF, UNKNOWN_SOURCE_NAME








Constructor Summary

Constructors 

Constructor and Description


BufferedTokenStream(TokenSource tokenSource) 









Method Summary

Methods 

Modifier and Type
Method and Description


protected int
adjustSeekIndex(int i)
Allowed derived classes to modify the behavior of operations which change
 the current stream position by adjusting the target token index of a seek
 operation.



void
consume()
Consumes the current symbol in the stream.



protected int
fetch(int n)
Add n elements to buffer.



void
fill()
Get all tokens from lexer until EOF.



protected List<Token>
filterForChannel(int from,
                int to,
                int channel) 


Token
get(int i)
Gets the Token at the specified index in the stream.



List<Token>
get(int start,
   int stop)
Get all tokens from start..stop inclusively.



List<Token>
getHiddenTokensToLeft(int tokenIndex)
Collect all hidden tokens (any off-default channel) to the left of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL.



List<Token>
getHiddenTokensToLeft(int tokenIndex,
                     int channel)
Collect all tokens on specified channel to the left of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL.



List<Token>
getHiddenTokensToRight(int tokenIndex)
Collect all hidden tokens (any off-default channel) to the right of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL
  or EOF.



List<Token>
getHiddenTokensToRight(int tokenIndex,
                      int channel)
Collect all tokens on specified channel to the right of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL or
  EOF.



String
getSourceName()
Gets the name of the underlying symbol source.



String
getText()
Get the text of all tokens in this buffer.



String
getText(Interval interval)
Return the text of all tokens within the specified interval.



String
getText(Object start,
       Object stop)
Return the text of all tokens in this stream between start and
 stop (inclusive).



String
getText(RuleContext ctx)
Return the text of all tokens in the source interval of the specified
 context.



List<Token>
getTokens() 


List<Token>
getTokens(int start,
         int stop) 


List<Token>
getTokens(int start,
         int stop,
         BitSet types)
Given a start and stop index, return a List of all tokens in
  the token type BitSet.



List<Token>
getTokens(int start,
         int stop,
         int ttype) 


TokenSource
getTokenSource()
Gets the underlying TokenSource which provides tokens for this
 stream.



int
index()
Return the index into the stream of the input symbol referred to by
 LA(1).



int
LA(int i)
Gets the value of the symbol at offset i from the current
 position.



protected void
lazyInit() 


protected Token
LB(int k) 


Token
LT(int k)
Get the Token instance associated with the value returned by
 LA(k).



int
mark()
A mark provides a guarantee that seek() operations will be
 valid over a "marked range" extending from the index where mark()
 was called to the current index().



protected int
nextTokenOnChannel(int i,
                  int channel)
Given a starting index, return the index of the next token on channel.



protected int
previousTokenOnChannel(int i,
                      int channel)
Given a starting index, return the index of the previous token on
 channel.



void
release(int marker)
This method releases a marked range created by a call to
 mark().



void
reset()
Deprecated. 
Use seek(0) instead.




void
seek(int index)
Set the input cursor to the position indicated by index.



void
setTokenSource(TokenSource tokenSource)
Reset this token stream by setting its token source.



protected void
setup() 


int
size()
Returns the total number of symbols in the stream, including a single EOF
 symbol.



protected boolean
sync(int i)
Make sure index i in tokens has a token.







Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait














Field Detail





tokenSource
@NotNull
protected TokenSource tokenSource
The TokenSource from which tokens for this stream are fetched.







tokens
protected List<Token> tokens
A collection of all tokens fetched from the token source. The list is
 considered a complete view of the input once fetchedEOF is set
 to true.







p
protected int p
The index into tokens of the current token (next token to
 consume()). tokens[p] should be
 LT(1).

 This field is set to -1 when the stream is first constructed or when
 setTokenSource(org.antlr.v4.runtime.TokenSource) is called, indicating that the first token has
 not yet been fetched from the token source. For additional information,
 see the documentation of IntStream for a description of
 Initializing Methods.







fetchedEOF
protected boolean fetchedEOF
Indicates whether the Token.EOF token has been fetched from
 tokenSource and added to tokens. This field improves
 performance for the following cases:

 
 consume(): The lookahead check in consume() to prevent
 consuming the EOF symbol is optimized by checking the values of
 fetchedEOF and p instead of calling LA(int).
 fetch(int): The check to prevent adding multiple EOF symbols into
 tokens is trivial with this field.
 









Constructor Detail





BufferedTokenStream
public BufferedTokenStream(@NotNull
                   TokenSource tokenSource)









Method Detail





getTokenSource
public TokenSource getTokenSource()
Description copied from interface: TokenStream
Gets the underlying TokenSource which provides tokens for this
 stream.

Specified by:
getTokenSource in interface TokenStream








index
public int index()
Description copied from interface: IntStream
Return the index into the stream of the input symbol referred to by
 LA(1).

 The behavior of this method is unspecified if no call to an
 initializing method has occurred after this stream was
 constructed.

Specified by:
index in interface IntStream








mark
public int mark()
Description copied from interface: IntStream
A mark provides a guarantee that seek() operations will be
 valid over a "marked range" extending from the index where mark()
 was called to the current index(). This allows the use of
 streaming input sources by specifying the minimum buffering requirements
 to support arbitrary lookahead during prediction.

 The returned mark is an opaque handle (type int) which is passed
 to release() when the guarantees provided by the marked
 range are no longer necessary. When calls to
 mark()/release() are nested, the marks must be released
 in reverse order of which they were obtained. Since marked regions are
 used during performance-critical sections of prediction, the specific
 behavior of invalid usage is unspecified (i.e. a mark is not released, or
 a mark is released twice, or marks are not released in reverse order from
 which they were created).

 The behavior of this method is unspecified if no call to an
 initializing method has occurred after this stream was
 constructed.

 This method does not change the current position in the input stream.

 The following example shows the use of mark(),
 release(mark), index(), and
 seek(index) as part of an operation to safely work within a
 marked region, then restore the stream position to its original value and
 release the mark.
  IntStream stream = ...;
 int index = -1;
 int mark = stream.mark();
 try {
   index = stream.index();
   // perform work here...
 } finally {
   if (index != -1) {
     stream.seek(index);
   }
   stream.release(mark);
 }
 

Specified by:
mark in interface IntStream
Returns:
An opaque marker which should be passed to
 release() when the marked range is no longer required.







release
public void release(int marker)
Description copied from interface: IntStream
This method releases a marked range created by a call to
 mark(). Calls to release() must appear in the
 reverse order of the corresponding calls to mark(). If a mark is
 released twice, or if marks are not released in reverse order of the
 corresponding calls to mark(), the behavior is unspecified.

 For more information and an example, see IntStream.mark().

Specified by:
release in interface IntStream
Parameters:
marker - A marker returned by a call to mark().
See Also:
IntStream.mark()







reset
@Deprecated
public void reset()
Deprecated. Use seek(0) instead.
This method resets the token stream back to the first token in the
 buffer. It is equivalent to calling seek(int)(0).
See Also:
setTokenSource(TokenSource)







seek
public void seek(int index)
Description copied from interface: IntStream
Set the input cursor to the position indicated by index. If the
 specified index lies past the end of the stream, the operation behaves as
 though index was the index of the EOF symbol. After this method
 returns without throwing an exception, then at least one of the following
 will be true.

 
   index() will return the index of the first symbol
     appearing at or after the specified index. Specifically,
     implementations which filter their sources should automatically
     adjust index forward the minimum amount required for the
     operation to target a non-ignored symbol.
   LA(1) returns IntStream.EOF
 

 This operation is guaranteed to not throw an exception if index
 lies within a marked region. For more information on marked regions, see
 IntStream.mark(). The behavior of this method is unspecified if no call to
 an initializing method has occurred after this stream
 was constructed.

Specified by:
seek in interface IntStream
Parameters:
index - The absolute index to seek to.







size
public int size()
Description copied from interface: IntStream
Returns the total number of symbols in the stream, including a single EOF
 symbol.

Specified by:
size in interface IntStream








consume
public void consume()
Description copied from interface: IntStream
Consumes the current symbol in the stream. This method has the following
 effects:

 
   Forward movement: The value of index()
                before calling this method is less than the value of index()
                after calling this method.
   Ordered lookahead: The value of LA(1) before
                calling this method becomes the value of LA(-1) after calling
                this method.
 

 Note that calling this method does not guarantee that index() is
 incremented by exactly 1, as that would preclude the ability to implement
 filtering streams (e.g. CommonTokenStream which distinguishes
 between "on-channel" and "off-channel" tokens).

Specified by:
consume in interface IntStream








sync
protected boolean sync(int i)
Make sure index i in tokens has a token.
Returns:
true if a token is located at index i, otherwise
    false.
See Also:
get(int i)







fetch
protected int fetch(int n)
Add n elements to buffer.
Returns:
The actual number of elements added to the buffer.







get
public Token get(int i)
Description copied from interface: TokenStream
Gets the Token at the specified index in the stream. When
 the preconditions of this method are met, the return value is non-null.

 The preconditions for this method are the same as the preconditions of
 IntStream.seek(int). If the behavior of seek(index) is
 unspecified for the current state and given index, then the
 behavior of this method is also unspecified.

 The symbol referred to by index differs from seek() only
 in the case of filtering streams where index lies before the end
 of the stream. Unlike seek(), this method does not adjust
 index to point to a non-ignored symbol.

Specified by:
get in interface TokenStream








get
public List<Token> get(int start,
              int stop)
Get all tokens from start..stop inclusively.







LA
public int LA(int i)
Description copied from interface: IntStream
Gets the value of the symbol at offset i from the current
 position. When i==1, this method returns the value of the current
 symbol in the stream (which is the next symbol to be consumed). When
 i==-1, this method returns the value of the previously read
 symbol in the stream. It is not valid to call this method with
 i==0, but the specific behavior is unspecified because this
 method is frequently called from performance-critical code.

 This method is guaranteed to succeed if any of the following are true:

 
   i>0
   i==-1 and index() returns a value greater
     than the value of index() after the stream was constructed
     and LA(1) was called in that order. Specifying the current
     index() relative to the index after the stream was created
     allows for filtering implementations that do not return every symbol
     from the underlying source. Specifying the call to LA(1)
     allows for lazily initialized streams.
   LA(i) refers to a symbol consumed within a marked region
     that has not yet been released.
 

 If i represents a position at or beyond the end of the stream,
 this method returns IntStream.EOF.

 The return value is unspecified if i<0 and fewer than -i
 calls to consume() have occurred from the beginning of
 the stream before calling this method.

Specified by:
LA in interface IntStream








LB
protected Token LB(int k)







LT
@NotNull
public Token LT(int k)
Description copied from interface: TokenStream
Get the Token instance associated with the value returned by
 LA(k). This method has the same pre- and post-conditions as
 IntStream.LA(int). In addition, when the preconditions of this method
 are met, the return value is non-null and the value of
 LT(k).getType()==LA(k).

Specified by:
LT in interface TokenStream
See Also:
IntStream.LA(int)







adjustSeekIndex
protected int adjustSeekIndex(int i)
Allowed derived classes to modify the behavior of operations which change
 the current stream position by adjusting the target token index of a seek
 operation. The default implementation simply returns i. If an
 exception is thrown in this method, the current stream index should not be
 changed.

 For example, CommonTokenStream overrides this method to ensure that
 the seek target is always an on-channel token.
Parameters:
i - The target token index.
Returns:
The adjusted target token index.







lazyInit
protected final void lazyInit()







setup
protected void setup()







setTokenSource
public void setTokenSource(TokenSource tokenSource)
Reset this token stream by setting its token source.







getTokens
public List<Token> getTokens()







getTokens
public List<Token> getTokens(int start,
                    int stop)







getTokens
public List<Token> getTokens(int start,
                    int stop,
                    BitSet types)
Given a start and stop index, return a List of all tokens in
  the token type BitSet.  Return null if no tokens were found.  This
  method looks at both on and off channel tokens.







getTokens
public List<Token> getTokens(int start,
                    int stop,
                    int ttype)







nextTokenOnChannel
protected int nextTokenOnChannel(int i,
                     int channel)
Given a starting index, return the index of the next token on channel.
 Return i if tokens[i] is on channel. Return the index of
 the EOF token if there are no tokens on channel between i and
 EOF.







previousTokenOnChannel
protected int previousTokenOnChannel(int i,
                         int channel)
Given a starting index, return the index of the previous token on
 channel. Return i if tokens[i] is on channel. Return -1
 if there are no tokens on channel between i and 0.

 
 If i specifies an index at or after the EOF token, the EOF token
 index is returned. This is due to the fact that the EOF token is treated
 as though it were on every channel.







getHiddenTokensToRight
public List<Token> getHiddenTokensToRight(int tokenIndex,
                                 int channel)
Collect all tokens on specified channel to the right of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL or
  EOF. If channel is -1, find any non default channel token.







getHiddenTokensToRight
public List<Token> getHiddenTokensToRight(int tokenIndex)
Collect all hidden tokens (any off-default channel) to the right of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL
  or EOF.







getHiddenTokensToLeft
public List<Token> getHiddenTokensToLeft(int tokenIndex,
                                int channel)
Collect all tokens on specified channel to the left of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL.
  If channel is -1, find any non default channel token.







getHiddenTokensToLeft
public List<Token> getHiddenTokensToLeft(int tokenIndex)
Collect all hidden tokens (any off-default channel) to the left of
  the current token up until we see a token on Lexer.DEFAULT_TOKEN_CHANNEL.







filterForChannel
protected List<Token> filterForChannel(int from,
                           int to,
                           int channel)







getSourceName
public String getSourceName()
Description copied from interface: IntStream
Gets the name of the underlying symbol source. This method returns a
 non-null, non-empty string. If such a name is not known, this method
 returns IntStream.UNKNOWN_SOURCE_NAME.

Specified by:
getSourceName in interface IntStream








getText
@NotNull
public String getText()
Get the text of all tokens in this buffer.

Specified by:
getText in interface TokenStream
Returns:
The text of all tokens in the stream.







getText
@NotNull
public String getText(Interval interval)
Description copied from interface: TokenStream
Return the text of all tokens within the specified interval. This
 method behaves like the following code (including potential exceptions
 for violating preconditions of TokenStream.get(int), but may be optimized by the
 specific implementation.

  TokenStream stream = ...;
 String text = "";
 for (int i = interval.a; i <= interval.b; i++) {
   text += stream.get(i).getText();
 }
 

Specified by:
getText in interface TokenStream
Parameters:
interval - The interval of tokens within this stream to get text
 for.
Returns:
The text of all tokens within the specified interval in this
 stream.







getText
@NotNull
public String getText(RuleContext ctx)
Description copied from interface: TokenStream
Return the text of all tokens in the source interval of the specified
 context. This method behaves like the following code, including potential
 exceptions from the call to TokenStream.getText(Interval), but may be
 optimized by the specific implementation.

 If ctx.getSourceInterval() does not return a valid interval of
 tokens provided by this stream, the behavior is unspecified.

  TokenStream stream = ...;
 String text = stream.getText(ctx.getSourceInterval());
 

Specified by:
getText in interface TokenStream
Parameters:
ctx - The context providing the source interval of tokens to get
 text for.
Returns:
The text of all tokens within the source interval of ctx.







getText
@NotNull
public String getText(Object start,
                     Object stop)
Description copied from interface: TokenStream
Return the text of all tokens in this stream between start and
 stop (inclusive).

 If the specified start or stop token was not provided by
 this stream, or if the stop occurred before the start
 token, the behavior is unspecified.

 For streams which ensure that the Token.getTokenIndex() method is
 accurate for all of its provided tokens, this method behaves like the
 following code. Other streams may implement this method in other ways
 provided the behavior is consistent with this at a high level.

  TokenStream stream = ...;
 String text = "";
 for (int i = start.getTokenIndex(); i <= stop.getTokenIndex(); i++) {
   text += stream.get(i).getText();
 }
 

Specified by:
getText in interface TokenStream
Parameters:
start - The first token in the interval to get text for.
stop - The last token in the interval to get text for (inclusive).
Returns:
The text of all tokens lying between the specified start
 and stop tokens.







fill
public void fill()
Get all tokens from lexer until EOF.
















Overview
Package
Class
Use
Tree
Deprecated
Index
Help




Prev Class
Next Class


Frames
No Frames


All Classes






Summary: 
Nested | 
Field | 
Constr | 
Method


Detail: 
Field | 
Constr | 
Method






Copyright © 1992–2019 Tunnel Vision Laboratories, LLC. All rights reserved.

Modifier and Type	Field and Description
`protected boolean`	`fetchedEOF` Indicates whether the `Token.EOF` token has been fetched from `tokenSource` and added to `tokens`.
`protected int`	`p` The index into `tokens` of the current token (next token to `consume()`).
`protected List<Token>`	`tokens` A collection of all tokens fetched from the token source.
`protected TokenSource`	`tokenSource` The `TokenSource` from which tokens for this stream are fetched.

Modifier and Type	Method and Description
`protected int`	`adjustSeekIndex(int i)` Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.
`void`	`consume()` Consumes the current symbol in the stream.
`protected int`	`fetch(int n)` Add `n` elements to buffer.
`void`	`fill()` Get all tokens from lexer until EOF.
`protected List<Token>`	`filterForChannel(int from, int to, int channel)`
`Token`	`get(int i)` Gets the `Token` at the specified `index` in the stream.
`List<Token>`	`get(int start, int stop)` Get all tokens from start..stop inclusively.
`List<Token>`	`getHiddenTokensToLeft(int tokenIndex)` Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on `Lexer.DEFAULT_TOKEN_CHANNEL`.
`List<Token>`	`getHiddenTokensToLeft(int tokenIndex, int channel)` Collect all tokens on specified channel to the left of the current token up until we see a token on `Lexer.DEFAULT_TOKEN_CHANNEL`.
`List<Token>`	`getHiddenTokensToRight(int tokenIndex)` Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on `Lexer.DEFAULT_TOKEN_CHANNEL` or EOF.
`List<Token>`	`getHiddenTokensToRight(int tokenIndex, int channel)` Collect all tokens on specified channel to the right of the current token up until we see a token on `Lexer.DEFAULT_TOKEN_CHANNEL` or EOF.
`String`	`getSourceName()` Gets the name of the underlying symbol source.
`String`	`getText()` Get the text of all tokens in this buffer.
`String`	`getText(Interval interval)` Return the text of all tokens within the specified `interval`.
`String`	`getText(Object start, Object stop)` Return the text of all tokens in this stream between `start` and `stop` (inclusive).
`String`	`getText(RuleContext ctx)` Return the text of all tokens in the source interval of the specified context.
`List<Token>`	`getTokens()`
`List<Token>`	`getTokens(int start, int stop)`
`List<Token>`	`getTokens(int start, int stop, BitSet types)` Given a start and stop index, return a `List` of all tokens in the token type `BitSet`.
`List<Token>`	`getTokens(int start, int stop, int ttype)`
`TokenSource`	`getTokenSource()` Gets the underlying `TokenSource` which provides tokens for this stream.
`int`	`index()` Return the index into the stream of the input symbol referred to by `LA(1)`.
`int`	`LA(int i)` Gets the value of the symbol at offset `i` from the current position.
`protected void`	`lazyInit()`
`protected Token`	`LB(int k)`
`Token`	`LT(int k)` Get the `Token` instance associated with the value returned by `LA(k)`.
`int`	`mark()` A mark provides a guarantee that `seek()` operations will be valid over a "marked range" extending from the index where `mark()` was called to the current `index()`.
`protected int`	`nextTokenOnChannel(int i, int channel)` Given a starting index, return the index of the next token on channel.
`protected int`	`previousTokenOnChannel(int i, int channel)` Given a starting index, return the index of the previous token on channel.
`void`	`release(int marker)` This method releases a marked range created by a call to `mark()`.
`void`	`reset()` Deprecated. Use `seek(0)` instead.
`void`	`seek(int index)` Set the input cursor to the position indicated by `index`.
`void`	`setTokenSource(TokenSource tokenSource)` Reset this token stream by setting its token source.
`protected void`	`setup()`
`int`	`size()` Returns the total number of symbols in the stream, including a single EOF symbol.
`protected boolean`	`sync(int i)` Make sure index `i` in tokens has a token.