RePrec::Collection - Parse relevance judgements for evaluation purposes
To do an evaluation of effectiveness of information retrieval methods one needs relevance judgements for queries and a collection under consideration. These need to be parsed for doing the evaluation. Class RePrec::Collection provides for means to do so which should suit for most formats of relevance judgments. In case it doesn't suit one can subclass this class. From a list of relevance judgements one needs to filter the query ID (QID), the document ID (DOCID) and a judgement (JUDGE) wether DOCID is relevant with respect to QID. As an additional parameter the number of documents in the collection under consideration is needed.
Constructor which does the parsing of a given judgements file. The constructor calls the private method
_init (with %parms as argument) in order to do the parsing. The argument %parms is described within the documentation of that method.
The file parsing method, which should be the only method to replace in subclasses of RePrec::Collection. Within this baseclass it is assumed that the data in $file comes as an table, with each row containing a QID, a DOCID and the judgement (JUDGE) itself. A document is marked relevant if the value of JUDGE equals 1. Argument %parms keep the following parameters (defaults are given in parens):
perl regular expression separating columns
column which holds the QIDs
column which holds the DOCIDs
column which holds the JUDGEs
perl regular expression; matching rows are ignored
number of documents in the collection under consideration.
returns 1 if document with ID $docid is relevant with respect to query with ID $qid. Else returns
returns number of documents with respect to the collection under consideration.
returns number of relevant documents for query with ID $qid with respect to the collection under consideration.
Yes. Please let me know!
Norbert Gövert <firstname.lastname@example.org>