String::Tokeniser - Perl extension for, uhm, tokenising strings.
String::Tokeniser provides an interface to a tokeniser class, allowing one to manipulate strings on a token-by-token basis without having to keep track of list element numbers and so on.
String::Tokeniser, tokenises $sentence and resets the token counter.
The next argument determines how a ``token'' is defined: a value of 0 or
undef determines that underscores are included in a token; -1 states that they are not. Alternatively, you can supply your own regular expression which will be fed to a
split to determine the tokens.
Then may optionally follow a list of exceptions: tokens that would be split in two, but should be treated as one.
Tells you if you have any more tokens left to deal with.
Move the `pointer' forward one (or
Return the current token; that is, the token under the `pointer'.
Return the previous token; that is, the one just past the `pointer'.
skiptoken;gettoken - the usual way of grabbing the next token in the list in turn.
Looks ahead one token, but does not change the `pointer' position.
Returns a string composed of the next
n tokens, but does not change the `pointer' position.
Assuming a string of tokens will end in
$string, returns everything from the current `pointer' position until the string is found. Returns a two-element list: firsly, why the search terminated, (either
EOF meaning we hit the end of the token list without success, or
$string was found.) and the rest of the tokens upto and including
$string (or the end of the list, whichever was soonest).
Saves one's pointer position. Can be used multiply as a save stack.
Restores a previously saved position.
At present, there is no support for exceptions which spread over three or more tokens, although this is planned.
Originaly written by Simon Cozens; Maintained by Alberto Simoes