The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.

Changes for version 0.03

  • few algorithm and documentation bugfixes
  • fixed updater script and library
  • new and updated data
  • now works with perl >= 5.8.5
  • INCOMPATIBLE CHANGE: tokens() method now takes threshold option which changes its behaviour
  • INCOMPATIBLE CHANGE: tokens_bounds() method doesn't store tokens anymore
  • disabled tests for now

Documentation

download newer data for tokenizer

Modules

tokenizer for OpenCorpora project
download newer data for tokenizer