CPTokenStream Class Reference
| Inherits from | NSObject |
| Declared in | CPTokenStream.h CPTokenStream.m |
Overview
CPTokenStreams store the output of a tokeniser ready for parsing.
This class manages a thread safe buffer between tokenising and parsing threads, blocking as apropriate to wait for new tokens. When a tokeniser has consumed its entire input it should call closeTokenStream to notify parsers that no further input will be found. Tokenisers that consume their entire input with no errors should produce a CPEOFToken at the end of the stream to inform parsers that the stream is complete.
Tasks
Creating and Initialising Token Streams
-
+ tokenStreamWithTokens:Creates a token stream with a set of tokens already ready for parsing.
-
– initWithTokens:Initialises a token stream with a set of tokens already ready for parsing.
Adding and Removing Tokens
-
– peekTokenReturns the first token in the stream but does not remove it from the stream.
-
– popTokenReturns the first token in the stream and removes it from the stream.
-
– pushToken:Adds a CPToken to the end of the token stream.
-
– pushTokens:Adds several CPTokens to the end of the token stream.
Finishing Tokenisation
-
– closeTokenStreamCloses the token stream, causing popToken to return
nilwhen all tokens have been exhausted rather than blocking waiting for new input.
Class Methods
tokenStreamWithTokens:
Creates a token stream with a set of tokens already ready for parsing.
+ (id)tokenStreamWithTokens:(NSArray *)tokensParameters
- tokens
A set of tokens to place at the start of the token stream.
Return Value
Returns a token stream containing tokens at its start.
Discussion
Creates a token stream with a set of tokens already ready for parsing.
See Also
Declared In
CPTokenStream.hInstance Methods
closeTokenStream
Closes the token stream, causing popToken to return nil when all tokens have been exhausted rather than blocking waiting for new input.
- (void)closeTokenStreamDiscussion
Closes the token stream, causing popToken to return nil when all tokens have been exhausted rather than blocking waiting for new input.
Declared In
CPTokenStream.hinitWithTokens:
Initialises a token stream with a set of tokens already ready for parsing.
- (id)initWithTokens:(NSArray *)tokensParameters
- tokens
A set of tokens to place at the start of the token stream.
Return Value
Returns the token stream, now containing tokens at its start.
Discussion
Initialises a token stream with a set of tokens already ready for parsing.
See Also
Declared In
CPTokenStream.hpeekToken
Returns the first token in the stream but does not remove it from the stream.
- (CPToken *)peekTokenReturn Value
The first CPToken in the stream.
Discussion
Returns the first token in the stream but does not remove it from the stream.
This method will not block waiting for a token to become available. If no token is available the method returns nil. If the token stream is closed the method returns nil.
See Also
Declared In
CPTokenStream.hpopToken
Returns the first token in the stream and removes it from the stream.
- (CPToken *)popTokenReturn Value
The first CPToken in the stream.
Discussion
Returns the first token in the stream and removes it from the stream.
This method will block waiting for a token to become available if the token stream is empty. If the token stream is closed the method returns nil.
See Also
Declared In
CPTokenStream.hpushToken:
Adds a CPToken to the end of the token stream.
- (void)pushToken:(CPToken *)tokenParameters
- token
The token to add to the stream.
Discussion
Adds a CPToken to the end of the token stream.
See Also
Declared In
CPTokenStream.hpushTokens:
Adds several CPTokens to the end of the token stream.
- (void)pushTokens:(NSArray *)tokensParameters
- tokens
The array of tokens to add to the token stream.
See Also
Declared In
CPTokenStream.h