NAME

Lingua::JA::NormalizeText - Text Normalizer

SYNOPSIS

  use Lingua::JA::NormalizeText;
  use utf8;

  my @options = ( qw/nfkc decode_entities/, \&dearinsu_to_desu );
  my $normalizer = Lingua::JA::NormalizeText->new(@options);

  print $normalizer->normalize('鳥が㌧㌦でありんす♥');
  # -> 鳥がトンドルです♥

  sub dearinsu_to_desu
  {
      my $text = shift;
      $text =~ s/でありんす/です/g;

      return $text;
  }

# or

  use Lingua::JA::NormalizeText qw/nfkc decode_entities/;
  use utf8;

  my $text = '㈱㋰㋫㋫♥';
  print decode_entities( nfkc($text) );
  # -> (株)ムフフ♥

DESCRIPTION

Lingua::JA::NormalizeText normalizes text.

METHODS

new(@options)

Creates a new Lingua::JA::NormalizeText instance.

The following options are available.

  OPTION                 SAMPLE INPUT        OUTPUT FOR SAMPLE INPUT
  ---------------------  ------------------  -----------------------
  lc                     DdD                 ddd
  uc                     DdD                 DDD
  nfkc                   ㌦                  ドル (length: 2)
  nfkd                   ㌦                  ドル (length: 3)
  nfc
  nfd
  decode_entities        ♥            ♥
  strip_html             <em>あ</em>             あ    
  alnum_z2h              ＡＢＣ１２３        ABC123
  alnum_h2z              ABC123              ＡＢＣ１２３
  space_z2h
  space_h2z
  katakana_z2h           ハァハァ            ﾊｧﾊｧ
  katakana_h2z           ｽｰﾊｰｽｰﾊｰ            スーハースーハー
  katakana2hiragana      パンツ              ぱんつ
  hiragana2katakana      ぱんつ              パンツ
  unify_3dots            はぁ。。。          はぁ…
  wave2tilde             〜                  ～
  tilde2wave             ～                  〜
  wavetilde2long         〜, ～              ー
  wave2long              〜                  ー
  tilde2long             ～                  ー
  fullminus2long         −                   ー
  dashes2long            —                   ー
  drawing_lines2long     ─                   ー
  unify_long_repeats     ヴァーーー          ヴァー
  nl2space               (new line)          (space)
  unify_long_spaces      (space)(space)      (space)
  remove_head_space      (space)あ(space)あ  あ(space)あ
  remove_tail_space      ああ(space)(space)  ああ
  old2new_kana           ゐヰゑヱ            いイえエ
  old2new_kanji          亞逸鬭              亜逸闘
  tab2space              (tab)(tab)          (space)(space)
  remove_controls        あ\x{0000}あ        ああ

The order in which these options are applied is according to the order of the elements of @options. (i.e., The first element is applied first, and the last element is applied last.)

External functions are also addable. (See dearinsu_to_desu function of SYNOPSIS section.)

remove_controls

Note that this option does not remove the following chars:

  CHARACTER TABULATION(tab)
  LINE FEED(LF)
  CARRIAGE RETURN(CR)

normalize($text)

normalizes $text.

AUTHOR

pawa <pawapawa@cpan.org>

SEE ALSO

新旧字体表: http://www.asahi-net.or.jp/~ax2s-kmtn/ref/old_chara.html

LICENSE

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.