Tony Bowden > Plucene > Plucene::Analysis::Standard::StandardTokenizer

Download:
Plucene-1.25.tar.gz

Dependencies

Annotate this POD

CPAN RT

New  12
Open  5
View/Report Bugs
Source  

NAME ^

Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

SYNOPSIS ^

        # isa Plucene::Analysis::CharTokenizer

DESCRIPTION ^

This is the standard tokenizer.

This should be a good tokenizer for most European-language documents.

METHODS ^

token_re

The regular expression for tokenising.

normalize

Remove 's and .

syntax highlighting: