Treex::Block::W2A::TokenizeOnWhitespace - Base tokenizer, splits on whitespaces, fills no_space_after
Each sentence is split into a sequence of tokens.
Flat a-tree is built and attributes
no_space_after are filled.
This class tokenizes only on whitespaces,
but it can be used as an ancestor for more apropriate tokenization by overriding the method
this method can be overridden in more advanced tokenizers
this method does all work of this tokenizer
David Mareček <firstname.lastname@example.org>
Martin Popel <email@example.com>
Copyright © 2011 by Institute of Formal and Applied Linguistics, Charles University in Prague
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.