Tomáš Kraut > Treex-Unilang > Treex::Block::W2A::TokenizeOnWhitespace

Download:
Treex-Unilang-0.08170.tar.gz

Dependencies

Annotate this POD

View/Report Bugs
Module Version: 0.08170   Source   Latest Release: Treex-Unilang-0.13095

NAME ^

Treex::Block::W2A::TokenizeOnWhitespace - Base tokenizer, splits on whitespaces, fills no_space_after

VERSION ^

version 0.08170

DESCRIPTION ^

Each sentence is split into a sequence of tokens. Flat a-tree is built and attributes no_space_after are filled. This class tokenizes only on whitespaces, but it can be used as an ancestor for more apropriate tokenization by overriding the method tokenize_sentence.

METHODS ^

tokenize_sentence()

this method can be overridden in more advanced tokenizers

process_zone()

this method does all work of this tokenizer

AUTHOR ^

David Mareček <marecek@ufal.mff.cuni.cz>

Martin Popel <popel@ufal.mff.cuni.cz>

COPYRIGHT AND LICENSE ^

Copyright © 2011 by Institute of Formal and Applied Linguistics, Charles University in Prague

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

syntax highlighting: