Data::Babel - Translator for biological identifiers
Version 1.03
use Data::Babel; use Data::Babel::Config; use Class::AutoDB; use DBI; # open database containing Babel metadata my $autodb=new Class::AutoDB(database=>'test'); # try to get existing Babel from database my $babel=old Data::Babel(name=>'test',autodb=>$autodb); unless ($babel) { # Babel does not yet exist, so we'll create it # idtypes, masters, maptables are names of configuration files that define # the Babel's component objects $babel=new Data::Babel (name=>'test',idtypes=>'examples/idtype.ini',masters=>'examples/master.ini', maptables=>'examples/maptable.ini'); } # open database containing real data my $dbh=DBI->connect("dbi:mysql:database=test",undef,undef); # translate several Entrez Gene ids to other types # CAUTION: rest of SYNOPSIS assumes you've loaded the real database somehow my $table=$babel->translate (dbh=>$dbh, input_idtype=>'gene_entrez', input_ids=>[1,2,3], output_idtypes=>[qw(gene_symbol gene_ensembl transcript_refseq transcript_ensembl chip_affy probe_affy chip_lumi probe_lumi)]); # print a few columns from each row of result for my $row (@$table) { print "Entrez gene=$row->[0]\tsymbol=$row->[1]\tEnsembl gene=$row->[2]\n"; } # generate a table mapping all Entrez Gene ids to UniProt ids my $table=$babel->translate (input_idtype=>'gene_entrez', input_ids_all=>1, output_idtypes=>[qw(protein_uniprot)]); # convert to HASH for easy programmatic lookups my %gene2uniprot=map {$_[0]=>$_[1]} @$table;
Data::Babel translates biological identifiers based on information contained in a database. Each Data::Babel object provides a unique mapping over a set of identifier types. The system as a whole can contain multiple Data::Babel objects; these may share some or all identifier types, and may provide the same or different mappings over the shared types.
The principal method is 'translate' which converts identifiers of one type into identifiers of one or more output types. In typical usage, you call 'translate' with a list of input ids to convert. You can also call it without any input ids (and with the special option 'input_ids_all' set) to generate a complete mapping of the input type to the output types. This is convenient if you want to hang onto the mapping for repeated use.
CAVEAT: Some features of Data::Babel are overly specific to the procedure we use to construct the underlying Babel database. We note such cases when they arise in the documentation below.
The main components of a Data::Babel object are
a master list of valid values for the type, and
optionally, a history mapping old values to current ones [NOT YET IMPLEMENTED]
One typically defines these components using configuration files whose basic format is defined in Config::IniFiles. See examples in "Configuration files" and the examples directory of the distribution.
Each MapTable represents a relational table stored in the database and provides a mapping over a subset of the Babel's IdTypes; the ensemble of MapTables must, of course, cover all the IdTypes. The ensemble of MapTables must also be non-redundant as explained in "Technical details".
You need not explicitly define Masters for all IdTypes; Babel will create 'implicit' Masters for any IdTypes lacking explicit ones. An implicit Master has a list of valid identifiers but no history and could be implemented as a view over all MapTables containing the IdType. In the current implementation, we use views for IdTypes contained in single MapTables but construct actual tables for IdTypes contained in multiple MapTables.
Our configuration files use 'ini' format as described in Config::IniFiles: 'ini' format files consist of a number of sections, each preceded with the section name in square brackets, followed by parameter names and their values.
There are separate config files for IdTypes, Masters, and MapTables. There are complete example files in the distribution. Here are some excerpts:
IdType
[chip_affy] display_name=Affymetrix array referent=array defdb=affy meta=name format=/^[a-z]+\d+/ sql_type=VARCHAR(32)
The section name is the IdType name. The parameters are
display_name. human readable name for this type
referent. the type of things to which this type of identifier refers
defdb. the database, if any, responsible for assigning this type of identifier
meta. some identifiers are purely synthetic (eg, Entrez gene IDs) while others have some mnemonic content; legal values are
eid (meaning synthetic)
symbol
name
description
format. Perl format of valid identifiers
sql_type. SQL data type
Master
[gene_entrez_master] inputs=<<INPUTS MainData/GeneInformation INPUTS query=<<QUERY SELECT locus_link_eid AS gene_entrez FROM gene_information QUERY
The section name is the Master name; the name of the IdType is the same but without the '_master'. The parameters are used by our database construction procedure and may not be useful in other settings.
MapTable
[gene_entrez_information] inputs=MainData/GeneInformation idtypes=gene_entrez gene_symbol gene_description organism_name_common query=<<QUERY SELECT GENE.locus_link_eid AS gene_entrez, GENE.symbol AS gene_symbol, GENE.description AS gene_description, ORG.common_name AS organism_name_common FROM gene_information AS GENE LEFT OUTER JOIN organism AS ORG ON GENE.organism_id=ORG.organism_id QUERY [% maptable %] inputs=MainData/GeneUnigene idtypes=gene_entrez gene_unigene query=<<QUERY SELECT UG.locus_link_eid AS gene_entrez, UG.unigene_eid AS gene_unigene FROM gene_unigene AS UG QUERY
This excerpt has two MapTable definitions which illustrate two ways that MapTables can be named. The first uses a normal section name; the second invokes a Template Toolkit macro which generates unique names of the form 'maptable_001'. This is very convenient because Babel databases typically contain a large number of MapTables, and it's hard to come up with good names for most of them. In any case, the names don't matter much, because software generates the queries that operate on these tables.
The 'inputs' and 'query' parameters are used by our database construction procedure and may not be useful in other settings.
The 'translate' method does not return any output for input identifiers that do not connect to any identifiers of the desired output types. In other words, 'translate' never returns output rows in which the output columns are all NULL.
An input identifier can fail to connect for several reasons:
If no output IdTypes are specified, 'translate' returns a row containing one element, namely, the input identifier, for each input id that exists in the corresponding Master table. This is the only way at present for the application to distinguish non-existent ids from ones that exist but don't connect.
A basic Babel property is that translations are stable. You can add output types to a query without changing the answer for the types you had before, you can remove output types from the query without changing the answer for the ones that remain, and if you "reverse direction" and swap the input type with one of the outputs, you get everything that was in the original answer.
We accomplish this by requiring that the database of MapTables satisfy the universal relation property (a well-known concept in relational database theory), and that 'translate' retrieves a sub-table of the universal relational. Concretely, the universal relational is the natural full outer join of all the MapTables. 'translate' performs natural left out joins starting with the Master table for the input IdType, and then including enough tables to connect the input and output IdTypes. Left outer joins suffice, because 'translate' starts with the Master.
We further require that the database of MapTables be non-redundant. The basic idea is that a given IdType may not be present in multiple MapTables, unless it is being used as join column. More technically, we require that the MapTables form a tree schema (another well-known concept in relational database theory), and any pair of MapTables have at most one IdType in common. As a consequence, there is essentially a single path between any pair of IdTypes.
To represent the connections between IdTypes and MapTables we use an undirected graph whose nodes represent IdTypes and MapTables, and whose edges go between each MapTable and the IdTypes it contains. In this representation, a non-redundant schema is a tree.
'translate' uses this graph to find the MapTables it must join to connect the input and output IdTypes. The algorithms is simple: start at the leaves and recursively prune back branches that do not contain the input or output IdTypes.
Title : new Usage : $babel=new Data::Babel name=>$name, idtypes=>$idtypes,masters=>$masters,maptables=>$maptables Function: Create new Data::Babel object or fetch existing object from database and update its components. Store the new or updated object. Returns : Data::Babel object Args : name eg, 'test' idtypes, masters, maptables define component objects; see below old existing Data::Babel object in case program already fetched it (typically via 'old') autodb Class::AutoDB object for database containing Babel. class method often set before running 'new' Notes : 'name' is required. All other args are optional
The component object parameters can be any of the following:
Title : old Usage : $babel=old Data::Babel($name) -- OR -- $babel=old Data::Babel(name=>$name) Function: Fetch existing Data::Babel object from database Returns : Data::Babel object or undef Args : name of Data::Babel object, eg, 'test' if keyword form used, can also specify autodb to set the corresponding class attribute
The available object attributes are
name eg, 'test' id name prefixed with 'babel', eg, 'babel:test'. not really used. exists for compatibility with component objects idtypes ARRAY of this Babel's Data::Babel::IdType objects masters ARRAY of this Babel's Data::Babel::Master objects maptables ARRAY of this Babel's Data::Babel::MapTable objects
The available class attributes are
autodb Class::AutoDB object for database containing Babel
Title : translate Usage : $table=$babel->translate (input_idtype=>'gene_entrez', input_ids=>[1,2,3], output_idtypes=>[qw(transcript_refseq transcript_ensembl)], limit=>100) -- OR -- $table=$babel->translate (input_idtype=>'gene_entrez', input_ids_all=>1, output_idtypes=>[qw(transcript_refseq transcript_ensembl)], limit=>100000) Function: Translate the input ids into ids of the output types Returns : table represented as an ARRAY of ARRAYS. Each inner ARRAY is one row of the result; the first element of each is an input id, the rest are outputs in the same order as output_idtypes Args : input_idtype name of Data::Babel::IdType object or object input_ids ARRAY of ids to be translated. It is an error to set both input_ids and input_ids_all. input_ids_all If true, all ids of the input type are translated. We recommend that 1 be used as the true value. See Notes below. It is an error to set both input_ids and input_ids_all. output_idtypes ARRAY of names of Data::Babel::IdType objects or objects limit maximum number of rows to retrieve (optional) Notes : Duplicate columns are retained. Does not return output rows in which the output columns are all NULL. If no output idtypes are specified, returns rows for which the input id exists in the corresponding Master table. The order of output rows is arbitrary. At present, input_ids_all may be set to any true value but in future, we may interpret the value differently. It is an error to set both input_ids and input_ids_all.
Title : show Usage : $babel->show Function: Print object in readable form Returns : nothing useful Args : none
Title : check_schema Usage : @errstrs=$babel->check_schema -- OR -- $ok=$babel->check_schema Function: Validate schema. Presently checks that schema graph is tree and all IdTypes contained in some MapTable Returns : in array context, list of errors in scalar context, true if schema is good, false if schema is bad Args : none
Title : check_contents Usage : $babel->check_schema Function: Validate contents of Babel database. Checks consistency of explicit Masters and MapTables Returns : boolean Args : none
Objects have names and ids: names are strings like 'gene_entrez' and are unique for a given class of object; ids have a short form of the type prepended to the name, eg, 'idtype:gene_entrez', and are unique across all classes. We use ids as nodes in schema and query graphs. In most cases, applications should should use names.
The methods in this section map names or ids to component objects, or (as a trivial convenience), convert ids to names.
Title : name2idtype Usage : $idtype=$babel->name2idtype('gene_entrez') Function: Get the IdType object given its name Returns : Data::Babel::IdType object or undef Args : name of object Notes : only looks at this Babel's component objects
Title : name2master Usage : $master=$babel->name2master('gene_entrez_master') Function: Get the Master object given its name Returns : Data::Babel::Master object or undef Args : name of object Notes : only looks at this Babel's component objects
Title : name2maptable Usage : $maptable=$babel->name2maptable('maptable_012') Function: Get the MapTable object given its name Returns : Data::Babel::MapTable object or undef Args : name of object Notes : only looks at this Babel's component objects
Title : id2object Usage : $object=$babel->id2object('idtype:gene_entrez') Function: Get object given its id Returns : Data::Babel::IdType, Data::Babel::Master, Data::Babel::MapTable object or undef Args : id of object Notes : only looks at this Babel's component objects
Title : id2name Usage : $name=$babel->id2name('idtype:gene_entrez') -- OR -- $name=Data::Babel->id2name('idtype:gene_entrez') Function: Convert object id to name Returns : string Args : id of object Notes : trival convenience method
Title : new Usage : $idtype=new Data::Babel::IdType name=>$name,... Function: Create new Data::Babel::IdType object or fetch existing object from database and update its components. Store the new or updated object. Returns : Data::Babel::IdType object Args : any attributes listed in the attributes section below, except 'id' (because it is computed from name) old existing Data::Babel object in case program already fetched it (typically via 'old') autodb Class::AutoDB object for database containing Babel. class method often set before running 'new' Notes : 'name' is required. All other args are optional
Title : old Usage : $idtype=old Data::Babel::IdType($name) -- OR -- $babel=old Data::Babel::IdType(name=>$name) Function: Fetch existing Data::Babel::IdType object from database Returns : Data::Babel::IdType object or undef Args : name of Data::Babel::IdType object, eg, 'gene_entrez' if keyword form used, can also specify autodb to set the corresponding class attribute
name eg, 'gene_entrez' id name prefixed with 'idtype', eg, 'idtype:::gene_entrez' master Data::Babel::Master object for this IdType maptables ARRAY of Data::Babel::MapTable objects containing this IdType display_name human readable name, eg, 'Entrez Gene ID' referent the type of things to which this type of identifier refers defdb the database, if any, which assigns identifiers meta meta-type: eid (meaning synthetic), symbol, name, description format Perl format of valid identifiers, eg, /^\d+$/ perl_format synonym for format sql_type SQL data type, eg, INT(11)
Title : degree Usage : $number=$idtype->degree Function: Tell how many Data::Babel::MapTables contain this IdType Returns : number Args : none
Title : new Usage : $master=new Data::Babel::Master name=>$name,idtype=>$idtype,... Function: Create new Data::Babel::Master object or fetch existing object from database and update its components. Store the new or updated object. Returns : Data::Babel::Master object Args : any attributes listed in the attributes section below, except 'id' (because it is computed from name) old existing Data::Babel object in case program already fetched it (typically via 'old') autodb Class::AutoDB object for database containing Babel. class method often set before running 'new' Notes : 'name' is required. All other args are optional
Title : old Usage : $master=old Data::Babel::Master($name) -- OR -- $babel=old Data::Babel::Master(name=>$name) Function: Fetch existing Data::Babel::Master object from database Returns : Data::Babel::Master object or undef Args : name of Data::Babel::Master object, eg, 'gene_entrez' if keyword form used, can also specify autodb to set the corresponding class attribute
name eg, 'gene_entrez_master' id name prefixed with 'master', eg, 'master:::gene_entrez_master' idtype Data::Babel::IdType object for which this is the Master implicit boolean indicating whether Master is implicit explicit opposite of implicit view boolean indicating whether Master is implemented as a view inputs, namespace, query used by our database construction procedure
Title : degree Usage : $number=$master->degree Function: Tell how many Data::Babel::MapTables contain this Master's IdType Returns : number Args : none
Title : new Usage : $maptable=new Data::Babel::MapTable name=>$name,idtypes=>$idtypes,... Function: Create new Data::Babel::MapTable object or fetch existing object from database and update its components. Store the new or updated object. Returns : Data::Babel::MapTable object Args : any attributes listed in the attributes section below, except 'id' (because it is computed from name) old existing Data::Babel object in case program already fetched it (typically via 'old') autodb Class::AutoDB object for database containing Babel. class method often set before running 'new' Notes : 'name' is required. All other args are optional
Title : old Usage : $maptable=old Data::Babel::MapTable($name) -- OR -- $babel=old Data::Babel::MapTable(name=>$name) Function: Fetch existing Data::Babel::MapTable object from database Returns : Data::Babel::MapTable object or undef Args : name of Data::Babel::MapTable object, eg, 'gene_entrez' if keyword form used, can also specify autodb to set the corresponding class attribute
name eg, 'gene_entrez_master' id name prefixed with 'maptable', eg, 'maptable:::gene_entrez_master' idtypes ARRAY of Data::Babel::IdType objects contained by this MapTable inputs, namespace, query used by our database construction procedure
I'm not aware of anything.
Nat Goodman, <natg at shore.net>
<natg at shore.net>
Please report any bugs or feature requests to bug-data-babel at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Data-Babel. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
bug-data-babel at rt.cpan.org
You can find documentation for this module with the perldoc command.
perldoc Data::Babel
You can also look for information at:
RT: CPAN's request tracker
http://rt.cpan.org/NoAuth/Bugs.html?Dist=Data-Babel
AnnoCPAN: Annotated CPAN documentation
http://annocpan.org/dist/Data-Babel
CPAN Ratings
http://cpanratings.perl.org/d/Data-Babel
Search CPAN
http://search.cpan.org/dist/Data-Babel/
This module extends a version developed by Victor Cassen.
Copyright 2010 Institute for Systems Biology
This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.
See http://dev.perl.org/licenses/ for more information.
To install Data::Babel, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Data::Babel
CPAN shell
perl -MCPAN -e shell install Data::Babel
For more information on module installation, please visit the detailed CPAN module installation guide.