Lingua::ZH::WordSegmenter - Simplified Chinese Word Segmentation
use Lingua::ZH::WordSegmenter; my $segmenter = Lingua::ZH::WordSegmenter->new(); print encode('gbk', $segmenter->seg($_) );
This is a perl version of simplified Chinese word segmentation.
The algorithm for this segmenter is to search the longest word at each point from both left and right directions, and choose the one with higher frequency product.
The original program is from the CPAN module Lingua::ZH::WordSegment (http://search.cpan.org/~chenyr/) I did the follwing changes: 1) make the interface object oriented; 2) make the internal string into utf8; 3) using sogou's dictionary (http://www.sogou.com/labs/dl/w.html) as the default dictionary.
This method constructs a new
Lingua::ZH::WordSegmenter object and returns it. Key/value pair arguments may be provided to set up the initial state. The following options correspond to attribute methods described below:
KEY PURPOSE DEFAULT ----------- ------------- -------------------- dic filename of the dic sogou dic dic_encoding encoding of the dic "gbk" seperator string to seperate wrods " " verbose show the segment process 0
Segment a input string, you can specify the encoding by the optional parameter.
The return result is encoded in utf8 format.
<jzhang533 at gmail.com>
Copyright 2007 Zhang Jun, all rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.