Francesco Nidito > Algorithm-WordLevelStatistics-0.03 > Algorithm::WordLevelStatistics



Annotate this POD

View/Report Bugs
Module Version: 0.03   Source  


Algorithm::WordLevelStatistics - Pure Perl implementation of the "words level statistics" algorithm


 use Algorithm::WordLevelStatistics;
 my $wls = Algorithm::WordLevelStatistics->new;
 my %spectra = (); # hash from word to positions
 open IN, "<file.txt" or die "cannot open file.txt: $!";
 my $idx = 0;
 while(<IN>) {
   next if(m/^\s*$/); #skip blank lines
   foreach my $w ( split /\W/, lc( $_ ) ) {
     next if($w =~ m/^\s*$/);
     push @{ $spectra{$w} }, $idx++;
 close IN;
 my $ws = $wls->compute_spectra( \%spectra );
 # sort the words by their C attribute (the deviation of sigma_nor with respect to the expected value in a random text)
 my @sw = sort { $ws->{$b}->{C} <=> $ws->{$a}->{C} } keys( %{ $ws } );
 # print all the words with their scores
 foreach my $i (@sw) {
   print $i, " => { C = ",$ws->{$i}->{C}, ", count = ", $ws->{$i}->{count}, ", sigma_nor = ", $ws->{$i}->{sigma_nor}," }\n";


This module implements the word leval statistics algorithm as described in: P. Carpena, P. Bernaola-Galav, M. Hackenberg, A.V. Coronado and J.L. Oliver, "Level statistics of words: finding keywords in literary texts and DNA", Physical Review E 79, 035102-4 ( DOI: 10.1103/PhysRevE.79.035102 )



Creates a new Algorithm::WordLevelStatistics object and returns it.


The return value is a reference to an hash of hashes like the following one:

   universe => {
                  C => 50.2020428972437,
                  count => 47,
                  sigma_nor => 6.16069263723295
   x => {
          C => 47.1009427722911,
          count => 150,
          sigma_nor => 3.71679156784182

The return value is a reference to an hash like the following one:

   C => 50.2020428972437,
   count => 47,
   sigma_nor => 6.16069263723295


The word level statistics algorithm uses a generalization of the level statistics analysis of quantum disordered systems to extract automatically keywords in literary texts.

The systems takes into account not only the relative frequencies of the words present in the text but also their spatial distribution in the text, and it is based on the consideration that relevant words are naturally clustered by the authors of the documents and irrelevant words are distributed randomly in the text (e.g. in an english text, the word 'the' is used with almost uniform distribution).

The word level statistics does not need a reference corpus but it uses just one document to extract the document's keywords. Moreover it is to be considered "language agnostic", because the algorithm does not need a "a priori" words classification (e.g. stop-words)



Corrected the test case (added test file to tarball.


Removed the dependency from Statistics::Lite. The removal of the dependency make this a self contained Perl module. The module is indeed faster too! (~15% speed improvement on large files).


Initial version of the module


Francesco Nidito


Copyright 2009 Francesco Nidito. All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.


syntax highlighting: