Tony Bowden > Plucene > Plucene::Analysis::Tokenizer

Download:
Plucene-1.25.tar.gz

Dependencies

Annotate this POD

CPAN RT

New  12
Open  5
View/Report Bugs
Source  

NAME ^

Plucene::Analysis::Tokenizer - base class for tokenizers

SYNOPSIS ^

        my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                ->new({ reader => $reader });

DESCRIPTION ^

This is an abstract base class for tokenizers.

A Tokenizer is a TokenStream whose input is a Reader.

METHODS ^

new

        my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                ->new({ reader => $reader });

This will create a new tokenizer.

next

This must be defined in a subclass

close

Close the input reader.

syntax highlighting: