Change from BSP version 0.3 to version 0.5
------------------------------------------
Satanjeev Banerjee, bane0025@d.umn.edu
Ted Pedersen, tpederse@d.umn.edu
University of Minnesota, Duluth
Program count.pl:
-----------------
1. Salient changes:
a. Support for n-grams, as opposed to only bigrams.
b. Support for user-defined token definitions.
c. Output format is different and explicitly demarcates tokens in
the ngrams. 'Extended' output is also supported (see below).
2. New switches in version 0.5:
a. --ngram => allows creation of ngrams with more than 2 tokens per
ngram. Version 0.3 only allowed bigrams.
b. --token => allows user definition of what a token should be for
creation of ngrams. Version 0.3 had two fixed regular
expressions: these are the defaults of version 0.5.
c. --set_freq_combo, --get_freq_combo => allows user control on the
number of frequency values to show per ngram. Version 0.3 only
allowed bigrams, and so there were only 3 values to be printed,
and no user control was allowed.
d. --remove => allows for the removal of ngrams that fall below a
given threshold. Scores dont show the ngrams that are removed.
e. --newLine => prevents ngram from spanning across the new line
character. Version 0.3 always considers the new line character
to be a white space character.
f. --extended => creates an 'extended' output where the values of
switches used are also output.
Program statistic.pl:
---------------------
1. Salient changes:
a. Support for n-grams, as opposed to only bigrams.
b. Performs error handling for library routines and allows routines
to throw errors.
2. New switches in version 0.5:
a. --ngram => allows user to specify the size of incoming ngrams.
b. --set_freq_combo FILE, --get_freq_combo => allows user to
specify what frequency values are available for the ngrams in
the incoming ngram list file.
c. --extended => informs statistic.pl that incoming ngram file may
have extended data in it and also requests statistic.pl to
create its own extended data in the lines of count.pl above.
Program rank.pl:
----------------
1. Salient changes:
a. Changes in inputs: This program no longer takes as input two
libraries and one frequency file to compute the rank correlation
coefficient. Instead it takes as input two files containing the
same ngrams sorted using two different statistical measures, and
then computes the rank correlation coefficient for the ngrams
that are pressent in both the files. using the two
files.
b. Changes in outputs: As before this program outputs the
correlation coefficient. Unlike before however the ngrams and
their ranks are not output since these are already available in
the input files.
2. Helper script:
We provide a helper script rank-script.sh to emulate the rank.pl
program of BSP version 0.3. See Readme.txt for more information on
this script.