Treex::Block::W2A::Tokenize - language independent rule based tokenizer
version 0.08170
Each sentence is split into a sequence of tokens using a series of regexs.
Flat a-tree is built and attributes no_space_after are filled.
This class uses language-independent regex rules for tokenization,
but it can be used as an ancestor for language-specific tokenization by overriding the method tokenize_sentence or by using around (see Moose::Manual::MethodModifiers).
David Mareček <marecek@ufal.mff.cuni.cz>
Martin Popel <popel@ufal.mff.cuni.cz>
Copyright © 2011 by Institute of Formal and Applied Linguistics, Charles University in Prague
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
