CPTokeniser Class Reference
| Inherits from | NSObject |
| Conforms to | NSCoding |
| Declared in | CPTokeniser.h CPTokeniser.m |
Overview
The CPTokeniser class provides tokenisation of NSStrings into CPTokenStreams, and describes what kinds of tokens to produce given particular string inputs.
Tokenisers are built up by adding a list of CPTokenRecogniser objects to the CPTokeniser. Each one recognises a different token. Each recogniser is given a chance to match a token in priority order. When a recogniser matches a token, the -tokeniser:shouldConsumeToken: delegate method is called. If this method returns NO, the rest of the recognisers are tried in priority order. If it returns YES, -tokeniser:willProduceToken: is called, and the resulting array of tokens added to the output stream.
Tasks
Managing the Delegate
-
delegateThe object that acts as a delegate to the receiving CPTokeniser.
property
Managing recognised tokens
-
– addTokenRecogniser:Adds a token recogniser at the end of the priority list of recognisers.
-
– insertTokenRecogniser:atPriority:Inserts a given token recogniser at a given priority level in the tokeniser.
-
– insertTokenRecogniser:beforeRecogniser:Inserts a given token recogniser before another.
-
– removeTokenRecogniser:Removes all occurances of recogniser in the tokeniser’s priority list.
Tokenising
-
– tokenise:Tokenises an input string by repeatedly using the recognisers in the tokeniser’s priority list.
-
– tokenise:into:Tokenises an input string into a pre-allocated output CPTokenStream.
Instance Methods
addTokenRecogniser:
Adds a token recogniser at the end of the priority list of recognisers.
- (void)addTokenRecogniser:(id<CPTokenRecogniser>)recogniserParameters
- recogniser
The token recogniser to add. This value must not be
nil.
Discussion
Adds a token recogniser at the end of the priority list of recognisers.
Exceptions
- Raises
an
NSInvalidArgumentExceptionif recogniser isnil.
Declared In
CPTokeniser.hinsertTokenRecogniser:atPriority:
Inserts a given token recogniser at a given priority level in the tokeniser.
- (void)insertTokenRecogniser:(id<CPTokenRecogniser>)recogniser atPriority:(NSInteger)priorityParameters
- recogniser
The token recogniser to insert. This value must not be
nil.
- priority
The priority level to insert at.
Discussion
Inserts a given token recogniser at a given priority level in the tokeniser.
The recogniser currently at that priority and all those below it move downwards.
Exceptions
- Raises
an
NSInvalidArgumentExceptionif recogniser isnil.
- Raises
an
NSRangeExceptionif priority is greater than the number of token recognisers in the tokeniser.
Declared In
CPTokeniser.hinsertTokenRecogniser:beforeRecogniser:
Inserts a given token recogniser before another.
- (void)insertTokenRecogniser:(id<CPTokenRecogniser>)recogniser beforeRecogniser:(id<CPTokenRecogniser>)otherParameters
- recogniser
The token recogniser to insert. This value must not be
nil.
- other
The token recogniser to insert before.
Discussion
Inserts a given token recogniser before another.
The recogniser currently at that priority and all those below it move downwards.
Exceptions
- Raises
an
NSInvalidArgumentExceptionif recogniser isnilor if other is not in the tokeniser’s priority queue.
Declared In
CPTokeniser.hremoveTokenRecogniser:
Removes all occurances of recogniser in the tokeniser’s priority list.
- (void)removeTokenRecogniser:(id<CPTokenRecogniser>)recogniserParameters
- recogniser
The token recogniser to remove.
Discussion
Removes all occurances of recogniser in the tokeniser’s priority list.
See Also
Declared In
CPTokeniser.htokenise:
Tokenises an input string by repeatedly using the recognisers in the tokeniser’s priority list.
- (CPTokenStream *)tokenise:(NSString *)inputParameters
- input
The input string to tokenise.
Return Value
Returns a token stream containing all tokens found in the input string.
Discussion
Tokenises an input string by repeatedly using the recognisers in the tokeniser’s priority list.
If the entire input is tokenised a CPEOFToken is added to the end of the result token stream. If not, the token stream ends with no EOF token.
Declared In
CPTokeniser.htokenise:into:
Tokenises an input string into a pre-allocated output CPTokenStream.
- (void)tokenise:(NSString *)input into:(CPTokenStream *)tokenStreamParameters
- input
The input string to tokenise.
- tokenStream
The token stream to add tokens to from the input string.
Discussion
Tokenises an input string into a pre-allocated output CPTokenStream.
If the entire input is tokenised a CPEOFToken is added to the end of the result token stream. If not, the token stream ends with no EOF token. This method can be useful for multithreading parsers, allowing you to create the token stream that the tokeniser writes to and pass it to both tokeniser and parser threads.
Declared In
CPTokeniser.h