report.pl - Summarize SenseClusters results with precision, recall, and confusion matrix
report.pl [OPTIONS] LABEL PRELABEL
report.pl --help for a quick summary of options
Reports the performance of discrimination in terms of the precision, recall and confusion table.
An output created by label.pl showing sense labels attached to the discovered clusters.
Sample LABEL files =>
C0 -> fine%5:00:00:elegant:00 C1 -> fine%3:00:00:: C2 -> fine%5:00:00:superior:02 C3 -> fine%5:00:00:satisfactory:00 C4 -> fine%5:00:00:thin:01
report will only read those lines from LABEL file that contain right arrow (->), all other lines will be ignored.
Lines containing '->' should show the cluster id on the left of the arrow and a sense tag on the right.
ClusterID -> SenseID 0 -> fine%5:00:00:elegant:00 1 -> fine%3:00:00:: 2 -> fine%5:00:00:superior:02 3 -> fine%5:00:00:satisfactory:00 4 -> fine%5:00:00:thin:01 Score = 60.00
Shows the actual output of label which contains a descriptive header line on the 1st line and the score of the mapping scheme on the last line.
Should be an output created by cluto2label.pl program showing the distribution of instances from each sense class in each of the clusters.
This distribution should be shown in a cluster by sense matrix where the rows represent the clusters and the columns represent the senses. Cell entry at CS[i][j] shows the number of instances belonging to cluster Ci that have the true true sense tag Sj.
0 //phone cord txt div form 0 2 1 0 0 0 1 2 4 0 5 2 2 25 4 0 1 9 0 0 0 1 0 1 0
Note that -
Displays this message.
Displays the version information.
Output will display a confusion table whose rows represent the discovered clusters and columns represent the actual sense classes such that cell value at (i,j) indicates the number of instances belonging to cluster Ci that have true sense id Sk where Sk is the column label on the top of the jth column. Columns are reordered such that the sense representing the rth column most accurately represents the rth cluster Cr and diagonal value at (r,r) shows the number of instances in the rth cluster that belong to their correct sense class.
When #clusters > #senses, clusters that aren't assigned a sense tag will have star (*) on them. When #senses > #clusters, senses that aren't assigned to any cluster will be hash (#) marked.
The sum of the diagonal entries shows the total number of instances that are correctly discriminated(#hits). From this number, report computes precision and recall where
precision = #hits / #clustered
#clustered = Number of instances clustered = total #instances - #instances that belong to the unlabelled clusters - #thrown shown in PRELABEL input file.
recall = #hits / #total instances
Sample Output :
S1 S2 S0 S3 TOTAL C0: 221 11 3 15 250 (5.71) C1: 295 395 448 144 1282 (29.28) C6: 430 233 441 68 1172 (26.77) C9: 145 44 149 105 443 (10.12) C2:* 0 1 135 2 138 (3.15) C3:* 138 4 4 2 148 (3.38) C4:* 0 0 182 0 182 (4.16) C5:* 2 6 150 6 164 (3.75) C7:* 41 159 99 97 396 (9.05) C8:* 0 0 203 0 203 (4.64) 1272 853 1814 439 4378 (29.05) (19.48) (41.43) (10.03) Precision = 36.92(1162/3147) Recall = 26.54(1162/4378+0) Legend of Sense Tags S0 = SERVE10 S1 = SERVE12 S2 = SERVE2 S3 = SERVE6
C1 represents S2 (stands for SERVE2), C6 represents S0 (stands for SERVE10) C9 represents S3 (stands for SERVE6)
Ted Pedersen, University of Minnesota, Duluth Amruta Purandare, University of Pittsburgh Anagha Kulkarni, Carnegie-Mellon University
Copyright (c) 2002-2008, Ted Pedersen, Amruta Purandare, Anagha Kulkarni
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to
The Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.