Datahub::Factory - A conveyor belt which transports data from a data source to a data sink.
dhconveyor [ARGUMENTS] [OPTIONS]
Datahub::Factory is a command line conveyor belt which automates three tasks:
Datahub::Factory fetches data from several sources as specified by the Importer settings, executes a Fix and sends it to a data sink, set by Exporter. Several importer and exporter modules are supported.
Datahub::Factory contains Log4perl support to monitor conveyor belt operations.
Note: This toolset is not a generic tool. It has been tailored towards the functional requirements of the Flemish Art Collection use case.
All commands share the following switches:
--log_level
Set the log_level. Takes a numeric parameter. Supported levels are: 1 (WARN), 2 (INFO), 3 (DEBUG). WARN (1) is the default.
--log_output
Selects an output for the log messages. By default, it will send them to STDERR (pass STDERR as parameter), but STDOUT (STDOUT) and a log file (logs/import_-date-.log) (STATISTICS) are also supported.
STDERR
STDOUT
logs/import_-date-.log
STATISTICS
Documentation about command line options.
It is possible to provide either all importer and/or exporter options on the command line, or to create a pipeline configuration file that sets those options.
Fetch data from a local or remote source, convert it to an exchange format and export the data.
Datahub::Factory uses a plugin-based architecture, making it easy to extend with new functionality.
New commands can be added by creating a Perl module that contains a command_name.pm file in the lib/Datahub/Factory/Command path. Datahub::Factory uses the Datahub::Factory::Command namespace and App::Cmd internally.
command_name.pm
lib/Datahub/Factory/Command
New Datahub::Factory::Importer, Exmporter and Fixer plugins can be added in the same way, in the lib/Datahub/Factory/Importer, Exporter or Fixer path. All plugins use the Datahub::Factory::Importer/Exporter/Fixer namespace and the namespace package as a Moose::Role.
lib/Datahub/Factory/Importer
Exporter
Fixer
Pieter De Praetere <pieter@packed.be>
Matthias Vandermaesen <matthias.vandermaesen@vlaamsekunstcollectie.be>
Copyright 2016 - PACKED vzw, Vlaamse Kunstcollectie vzw
This library is free software; you can redistribute it and/or modify it under the terms of the GPLv3.
To install Datahub::Factory, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Datahub::Factory
CPAN shell
perl -MCPAN -e shell install Datahub::Factory
For more information on module installation, please visit the detailed CPAN module installation guide.