org.antlr.runtime
Class CommonTokenStream

java.lang.Object
  extended by org.antlr.runtime.BufferedTokenStream
      extended by org.antlr.runtime.CommonTokenStream
All Implemented Interfaces:
IntStream, TokenStream
Direct Known Subclasses:
TokenRewriteStream

public class CommonTokenStream
extends BufferedTokenStream

The most common stream of tokens where every token is buffered up and tokens are filtered for a certain channel (the parser will only see these tokens). Even though it buffers all of the tokens, this token stream pulls tokens from the tokens source on demand. In other words, until you ask for a token using consume(), LT(), etc. the stream does not pull from the lexer. The only difference between this stream and BufferedTokenStream superclass is that this stream knows how to ignore off channel tokens. There may be a performance advantage to using the superclass if you don't pass whitespace and comments etc. to the parser on a hidden channel (i.e., you set $channel instead of calling skip() in lexer rules.)

See Also:
UnbufferedTokenStream, BufferedTokenStream

Field Summary
protected  int channel
          Skip tokens on any channel but this one; this is how we skip whitespace...
 
Fields inherited from class org.antlr.runtime.BufferedTokenStream
lastMarker, p, range, tokens, tokenSource
 
Constructor Summary
CommonTokenStream()
           
CommonTokenStream(TokenSource tokenSource)
           
CommonTokenStream(TokenSource tokenSource, int channel)
           
 
Method Summary
 void consume()
          Always leave p on an on-channel token.
 int getNumberOfOnChannelTokens()
          Count EOF just once.
protected  Token LB(int k)
           
 Token LT(int k)
          Get Token at current input pointer + i ahead where i=1 is next Token.
 void reset()
           
 void setTokenSource(TokenSource tokenSource)
          Reset this token stream by setting its token source.
protected  void setup()
           
protected  int skipOffTokenChannels(int i)
          Given a starting index, return the index of the first on-channel token.
protected  int skipOffTokenChannelsReverse(int i)
           
 
Methods inherited from class org.antlr.runtime.BufferedTokenStream
fetch, fill, get, get, getSourceName, getTokens, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LA, mark, range, release, rewind, rewind, seek, size, sync, toString, toString, toString
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

channel

protected int channel
Skip tokens on any channel but this one; this is how we skip whitespace...

Constructor Detail

CommonTokenStream

public CommonTokenStream()

CommonTokenStream

public CommonTokenStream(TokenSource tokenSource)

CommonTokenStream

public CommonTokenStream(TokenSource tokenSource,
                         int channel)
Method Detail

consume

public void consume()
Always leave p on an on-channel token.

Specified by:
consume in interface IntStream
Overrides:
consume in class BufferedTokenStream

LB

protected Token LB(int k)
Overrides:
LB in class BufferedTokenStream

LT

public Token LT(int k)
Description copied from interface: TokenStream
Get Token at current input pointer + i ahead where i=1 is next Token. i<0 indicates tokens in the past. So -1 is previous token and -2 is two tokens ago. LT(0) is undefined. For i>=n, return Token.EOFToken. Return null for LT(0) and any index that results in an absolute address that is negative.

Specified by:
LT in interface TokenStream
Overrides:
LT in class BufferedTokenStream

skipOffTokenChannels

protected int skipOffTokenChannels(int i)
Given a starting index, return the index of the first on-channel token.


skipOffTokenChannelsReverse

protected int skipOffTokenChannelsReverse(int i)

reset

public void reset()
Overrides:
reset in class BufferedTokenStream

setup

protected void setup()
Overrides:
setup in class BufferedTokenStream

getNumberOfOnChannelTokens

public int getNumberOfOnChannelTokens()
Count EOF just once.


setTokenSource

public void setTokenSource(TokenSource tokenSource)
Reset this token stream by setting its token source.

Overrides:
setTokenSource in class BufferedTokenStream


Copyright © 2011. All Rights Reserved.