CPAN RT

New	12
Open	4

Source Latest Release: Plucene-1.25

NAME

Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

        # isa Plucene::Analysis::CharTokenizer

This is the standard tokenizer.

This should be a good tokenizer for most European-language documents.

The regular expression for tokenising.