Tony Bowden > Plucene > Plucene::Analysis::CharTokenizer

Download:
Plucene-1.25.tar.gz

Dependencies

Annotate this POD

CPAN RT

New  12
Open  5
View/Report Bugs
Source  

NAME ^

Plucene::Analysis::CharTokenizer - base class for character tokenisers

SYNOPSIS ^

        # isa Plucene::Analysis::Tokenizer

        my $next = $chartokenizer->next;

DESCRIPTION ^

This is an abstract base class for simple, character-oriented tokenizers.

METHODS ^

token_re

This should be defined in subclasses.

normalize

This will normalise the character before it is added to the token.

next

        my $next = $chartokenizer->next;

This will return the next token in the string, or undef at the end of the string.

syntax highlighting: