Alberto Manuel Brandão Simões > Lingua-Identify-0.55 > Lingua::Identify

Download:
Lingua/Lingua-Identify-0.55.tar.gz

Dependencies

Annotate this POD

Related Modules

Win32::DriveInfo
HTML::Mason
Text::Soundex
Text::Metaphone
LWP::UserAgent
Thread::Queue
more...
By perlmonks.org

CPAN RT

New  1
Open  1
View/Report Bugs
Module Version: 0.55   Source   Latest Release: Lingua-Identify-0.57_1

NAME ^

Lingua::Identify - Language identification

SYNOPSIS ^

  use Lingua::Identify qw(:language_identification);
  $a = langof($textstring); # gives the most probable language

or the complete way:

  @a = langof($textstring); # gives pairs of languages / probabilities
                            # sorted from most to least probable

  %a = langof($textstring); # gives a hash of language / probability

or the expert way (see section OPTIONS, under HOW TO PERFORM IDENTIFICATION)

  $a = langof( { method => [qw/smallwords prefix2 suffix2/] }, $text);

  $a = langof( { 'max-size' => 3_000_000 }, $text);

  $a = langof( { 'extract_from' => ( 'head' => 1, 'tail' => 2)}, $text);

DESCRIPTION ^

STARTING WITH VERSION 0.25, Lingua::Identify IS UNICODE BY DEFAULT!

Lingua::Identify identifies the language a given string or file is written in.

See section WHY LINGUA::IDENTIFY for a list of Lingua::Identify's strong points.

See section KNOWN LANGUAGES for a list of available languages and HOW TO PERFORM IDENTIFICATION to know how to really use this module.

If you're in a hurry, jump to section EXAMPLES, way down below.

Also, don't forget to read the following section, IMPORTANT WARNING.

A WARNING ON THE ACCURACY OF LANGUAGE IDENTIFICATION METHODS ^

Take a word that exists in two different languages, take a good look at it and answer this question: "What language does this word belong to?".

You can't give an answer like "Language X", right? You can only say it looks like any of a set of languages.

Similarly, it isn't always easy to identify the language of a text if the only two active languages are very similar.

Now that we've taken out of the way the warning that language identification is not 100% accurate, please keep reading the documentation.

WHY LINGUA::IDENTIFY ^

You might be wondering why you should use Lingua::Identify instead of any other tool for language identification.

Here's a list of Lingua::Identify's strong points:

HOW TO PERFORM IDENTIFICATION ^

langof

To identify the language a given text is written in, use the langof function. To get a single value, do:

  $language = langof($text);

To get the most probable language and also the percentage of its probability, do:

  ($language, $probability) = langof($text);

If you want a hash where each active language is mapped into its percentage, use this:

  %languages = langof($text);

OPTIONS

langof can also be given some configuration parameters, in this way:

  $language = langof(\%config, $text);

These parameters are detailed here:

langof_file

langof_file works just like langof, with the exception that it reveives filenames instead of text. It reads these texts (if existing and readable, of course) and parses its content.

Currently, langof_file assumes the files are regular text. This may change in the future and the files might be scanned to check their filetype and then parsed to extract only their textual content (which should be pretty useful so that you can perform language identification, say, in HTML files, or PDFs).

To identify the language a file is written in:

  $language = langof_file($path);

To get the most probable language and also the percentage of its probability, do:

  ($language, $probability) = langof_file($path);

If you want a hash where each active language is mapped into its percentage, use this:

  %languages = langof_file($path);

If you pass more than one file to langof_file, they will all be read and their content merged and then parsed for language identification.

OPTIONS

langof_file accepts all the options langof does, so refer to those first (up in this document).

  $language = langof_file(\%config, $path);

langof_file currently only reads the first 10,000 bytes of each file.

You can force an input encoding with { encoding => 'ISO-8859-1' } in the configuration hash.

confidence

After getting the results into an array, its first element is the most probable language. That doesn't mean it is very probable or not.

You can find more about the likeliness of the results to be accurate by computing its confidence level.

  use Lingua::Identify qw/:language_identification/;
  my @results = langof($text);
  my $confidence_level = confidence(@results);
  # $confidence_level now holds a value between 0.5 and 1; the higher that
  # value, the more accurate the results seem to be

The formula used is pretty simple: p1 / (p1 + p2) , where p1 is the probability of the most likely language and p2 is the probability of the language which came in second. A couple of examples to illustrate this:

English 50% Portuguese 10% ...

confidence level: 50 / (50 + 10) = 0.83

Another example:

Spanish 30% Portuguese 10% ...

confidence level: 30 / (25 + 30) = 0.55

French 10% German 5% ...

confidence level: 10 / (10 + 5) = 0.67

As you can see, the first example is probably the most accurate one. Are there any doubts? The English language has five times the probability of the second language.

The second example is a bit more tricky. 55% confidence. The confidence level is always above 50%, for obvious reasons. 55% doesn't make anyone confident in the results, and one shouldn't be, with results such as these.

Notice the third example. The confidence level goes up to 67%, but the probability of French is of mere 10%. So what? It's twice as much as the second language. The low probability may well be caused by a great number of languages in play.

get_all_methods

Returns a list comprised of all the available methods for language identification.

LANGUAGE IDENTIFICATION IN GENERAL ^

Language identification is based in patterns.

In order to identify the language a given text is written in, we repeat a given process for each active language (see section LANGUAGES MANIPULATION); in that process, we look for common patterns of that language. Those patterns can be prefixes, suffixes, common words, ngrams or even sequences of words.

After repeating the process for each language, the total score for each of them is then used to compute the probability (in percentage) for each language to be the one of that text.

METHODS OF LANGUAGE IDENTIFICATION ^

Lingua::Identify currently comprises four different ways for language identification, in a total of thirteen variations of those.

The available methods are the following: smallwords, prefixes1, prefixes2, prefixes3, prefixes4, suffixes1, suffixes2, suffixes3, suffixes4, ngrams1, ngrams2, ngrams3 and ngrams4.

Here's a more detailed explanation of each of those ways and those methods

Small Word Technique - smallwords

The "Small Word Technique" searches the text for the most common words of each active language. These words are usually articles, pronouns, etc, which happen to be (usually) the shortest words of the language; hence, the method name.

This is usually a good method for big texts, especially if you happen to have few languages active.

Prefix Analysis - prefixes1, prefixes2, prefixes3, prefixes4

This method analyses text for the common prefixes of each active language.

The methods are, respectively, for prefixes of size 1, 2, 3 and 4.

Suffix Analysis - suffixes1, suffixes2, suffixes3, suffixes4

Similar to the Prefix Analysis (see above), but instead analysing common suffixes.

The methods are, respectively, for suffixes of size 1, 2, 3 and 4.

Ngram Categorization - ngrams1, ngrams2, ngrams3, ngrams4

Ngrams are sequences of tokens. You can think of them as syllables, but they are also more than that, as they are not only comprised by characters, but also by spaces (delimiting or separating words).

Ngrams are a very good way for identifying languages, given that the most common ones of each language are not generally very common in others.

This is usually the best method for small amounts of text or too many active languages.

The methods are, respectively, for ngrams of size 1, 2, 3 and 4.

LANGUAGE MANIPULATION ^

When trying to perform language identification, Lingua::Identify works not with all available languages, but instead with the ones that are active.

By default, all available languages are active, but that can be changed by the user.

For your convenience, several methods regarding language manipulation were created. In order to use them, load the module with the tag :language_manipulation.

These methods work with the two letters code for languages.

activate_language

Activate a language

  activate_language('en');

  # or

  activate_language($_) for get_all_languages();
activate_all_languages

Activates all languages

  activate_all_languages();
deactivate_language

Deactivates a language

  deactivate_language('en');
deactivate_all_languages

Deactivates all languages

  deactivate_all_languages();
get_all_languages

Returns the names of all available languages

  my @all_languages = get_all_languages();
get_active_languages

Returns the names of all active languages

  my @active_languages = get_active_languages();
get_inactive_languages

Returns the names of all inactive languages

  my @active_languages = get_inactive_languages();
is_active

Returns the name of the language if it is active, an empty list otherwise

  if (is_active('en')) {
    # YOUR CODE HERE
  }
is_valid_language

Returns the name of the language if it exists, an empty list otherwise

  if (is_valid_language('en')) {
    # YOUR CODE HERE
  }
set_active_languages

Sets the active languages

  set_active_languages('en', 'pt');

  # or

  set_active_languages(get_all_languages());
name_of

Given the two letter tag of a language, returns its name

  my $language_name = name_of('pt');

KNOWN LANGUAGES ^

Currently, Lingua::Identify knows the following languages (33 total):

AF - Afrikaans
BG - Bulgarian
BR - Breton
BS - Bosnian
CY - Welsh
DA - Danish
DE - German
EN - English
EO - Esperanto
ES - Spanish
FI - Finnish
FR - French
FY - Frisian
GA - Irish
HR - Croatian
HU - Hungarian
ID - Indonesian
IS - Icelandic
IT - Italian
LA - Latin
MS - Malay
NL - Dutch
NO - Norwegian
PL - Polish
PT - Portuguese
RO - Romanian
RU - Russian
SL - Slovene
SO - Somali
SQ - Albanian
SV - Swedish
SW - Swahili
TR - Turkish

CONTRIBUTING WITH NEW LANGUAGES ^

Please do not contribute with modules you made yourself. It's easier to contribute with unprocessed text, because that allows for new versions of Lingua::Identify not having to drop languages down in case I can't contact you by that time.

Use make-lingua-identify-language to create a new module for your own personal use, if you must, but try to contribute with unprocessed text rather than those modules.

EXAMPLES ^

THE BASIC EXAMPLE

Check the language a given text file is written in:

  use Lingua::Identify qw/langof/;

  my $text = join "\n", <>;

  # identify the language by letting the module decide on the best way
  # to do so
  my $language = langof($text);

IDENTIFYING BETWEEN TWO LANGUAGES

Check the language a given text file is written in, supposing you happen to know it's either Portuguese or English:

  use Lingua::Identify qw/langof set_active_languages/;
  set_active_languages(qw/pt en/);

  my $text = join "\n", <>;

  # identify the language by letting the module decide on the best way
  # to do so
  my $language = langof($text);

TO DO ^

ACKNOWLEDGMENTS ^

The following people and/or projects helped during this tool development:

   * EuroParl v5 corpus was used to train Dutch, German, English,
     Spanish, Finish, French, Italian, Portuguese, Danish and Swedish.

SEE ALSO ^

langident(1), Text::ExtractWords(3), Text::Ngram(3), Text::Affixes(3).

ISO 639 Language Codes, at http://www.w3.org/WAI/ER/IG/ert/iso639.htm

AUTHOR ^

Alberto Simoes, <ambs@cpan.org>

Jose Castro, <cog@cpan.org>

COPYRIGHT & LICENSE ^

Copyright 2008-2010 Alberto Simoes, All Rights Reserved. Copyright 2004-2008 Jose Castro, All Rights Reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

syntax highlighting: