setup.pl
A Perl script that preprocesses and prepares DATA for Demo experiments with SenseClusters.
setup.pl [Options] DATA
Type 'setup.pl --help' for quick summary of the Options.
SenseClusters requires input DATA in Senseval-2 format. DATA in any other format has to be first converted to this format. We provide with this distribution a pre-processing program Toolkit/preprocess/plain/text2sval.pl that converts data in plain text format (with the context of single instance on each line) into Senseval-2 format.
SenseClusters uses an unsupervised clustering approach and hence doesn't require DATA to be sense tagged at all. However, if the true sense classes of the DATA instances are available, those could be used for evaluation.
Data Options specify the input DATA to setup.pl. There are three different possibilities, which can be denoted via the following regex:
setup.pl DATA (--training TRAIN or --split P)?
1. If neither --training nor --split options are used, DATA will be clustered using the features extracted from the same DATA file.
2. If a separate Training file is provided via --training TRAIN option, DATA will be clustered using features extracted from the given TRAIN file. TRAIN file is expected to be in the Senseval-2 format.
3. If DATA file is provided with the --split P option, (100-P)% of DATA will be clustered using features extracted from the rest P% DATA.
Thus, DATA can be provided in a single DATA file, or with the --split P option or with a separate training file via --training TRAIN option. Options --split and --training can't be both used together.
If the correct answer tags of the DATA instances are known, these can be used for performing some special tasks like evaluating results or filtering low frequency senses.
Specifies the true sense tags of the DATA instances.
Each line in the KEY file should show
<instance id=\"I\"\/> [<sense id=\"S\"\/>]+
where an Instance-Id is followed by its true sense tag/s.
KEY file in any other format has to be first converted to this format. We provide with this distribution a script called keyconvert.pl (in Toolkit/preprocess/sval2) to convert a KEY file in Senseval-2 format (such as fine.key) to SenseClusters' KEY format.
If the KEY file is not specified and if any sense-tag options like evaluation or sense-filter are used in wrapper discriminate.pl, SenseClusters assumes that the sense ids are embedded in the same DATA file and these will be searched by matching an expression
/sense\s*id=\"SID\"/
assuming that SID shows a true sense id of an immediately preceding instance ID matched by /instance id=\"ID\"/ expression.
TOKENFILE is a file of Perl regexes that define tokenization scheme in DATA. A sample regex file, token.regex is provided with this distribution and if the user doesn't specify the TOKENFILE via --token option, token.regex will be searched in the current directory.
NONTOKENFILE is a file of Perl regexes that define strings that will be removed prior to tokenization. If NONTOKENFILE is not specified, only those string sequences that are not tokens will be removed.
Displays to STDERR the current program status. Silent by default.
Displays to STDOUT values of required and option arguments.
Displays this message.
Displays the version information.
setup.pl preprocesses given DATA in following ways -
A test XML file containing instances of single 'LEXELT' from a given DATA file. If --split P is specified, LEXELT-test.xml will have (1-P)% of the instances in DATA with the value of <lexelt> tag as LEXELT. Otherwise, will have all DATA instances that come under the LEXELT item.
A count file containing instance data within <context> and </context> tags on a single line for each instance appearing in corresponding LEXELT-test.xml.
This file is created only if --training or --split options are used. If --split P is used, this will contain P% of the DATA instances having <lexelt> tag value = LEXELT. Otherwise, this file will have all instances from TRAINING that come under the LEXELT tag.
This is created only if its equivalent LEXELT-training.xml is created and contains instance data within <context> </context> tags on a single line for each instance appearing in LEXELT-training.xml.
Additionally, LexSample directory will have files token.regex and nontoken.regex if --nontoken option is used.
setup.pl uses a preprocessing program preprocess.pl which is included in SenseClusters
Ted Pedersen, University of Minnesota, Duluth
Amruta Purandare, University of Pittsburgh
The perldoc man page of the wrapper discriminate.pl that uses the training and test files created by setup.pl
Copyright (c) 2002-2006,
Ted Pedersen, University of Minnesota, Duluth. tpederse@umn.edu
Amruta Purandare, University of Pittsburgh. amruta@cs.pitt.edu
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to
The Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
