HPC::Runner::Command - Create composable bioinformatics hpc analyses.
To create a new project
hpcrunner.pl new MyNewProject
To submit jobs to a cluster
hpcrunner.pl submit_jobs
To run jobs on an interactive queue or workstation
hpcrunner.pl execute_job
HPC::Runner::Command is a set of libraries for scaffolding data analysis projects, submitting and executing jobs on an HPC cluster or workstation, and obsessively logging results.
Please see the complete documentation at HPC::Runner::Command GitBooks
You can create a new project, with a sane directory structure by using
Our simplest example is a single job type with no dependencies - each task is independent of all other tasks.
#preprocess.sh echo "preprocess" && sleep 10; echo "preprocess" && sleep 10; echo "preprocess" && sleep 10;
hpcrunner.pl submit_jobs --infile preprocess.sh
tree hpc-runner
Most of the time we have jobs that depend upon other jobs.
#blastx.sh #HPC jobname=unzip unzip Sample1.zip unzip Sample2.zip unzip Sample3.zip #HPC jobname=blastx #HPC deps=unzip blastx --db env_nr --sample Sample1.fasta blastx --db env_nr --sample Sample2.fasta blastx --db env_nr --sample Sample3.fasta
Within a job type we can declare dependencies on particular tasks.
#blastx.sh #HPC jobname=unzip #TASK tags=Sample1 unzip Sample1.zip #TASK tags=Sample2 unzip Sample2.zip #TASK tags=Sample3 unzip Sample3.zip #HPC jobname=blastx #HPC deps=unzip #TASK tags=Sample1 blastx --db env_nr --sample Sample1.fasta #TASK tags=Sample2 blastx --db env_nr --sample Sample2.fasta #TASK tags=Sample3 blastx --db env_nr --sample Sample3.fasta
Each scheduler has its own set of variables. HPC::Runner::Command has a set of generalized variables for declaring types across templates. For more information please see Job Scheduler Comparison
Additionally, for workflows with a large number of tasks, please see Considerations for Workflows with a Large Number of Tasks for information on how to group tasks together.
#blastx.sh #HPC jobname=unzip #HPC cpus_per_task=1 #HPC partition=serial #HPC commands_per_node=1 #TASK tags=Sample1 unzip Sample1.zip #TASK tags=Sample2 unzip Sample2.zip #TASK tags=Sample3 unzip Sample3.zip #HPC jobname=blastx #HPC cpus_per_task=6 #HPC deps=unzip #TASK tags=Sample1 blastx --threads 6 --db env_nr --sample Sample1.fasta #TASK tags=Sample2 blastx --threads 6 --db env_nr --sample Sample2.fasta #TASK tags=Sample3 blastx --threads 6 --db env_nr --sample Sample3.fasta
Jillian Rowe <jillian.e.rowe@gmail.com>
This software was previously released under HPC::Runner. HPC::Runner::Command is a complete rewrite of the existing library. While it is meant to have much of the same functionality, it is not backwords compatible.
As of Version 2.41:
This modules continuing development is supported by NYU Abu Dhabi in the Center for Genomics and Systems Biology. With approval from NYUAD, this information was generalized and put on bitbucket, for which the authors would like to express their gratitude.
Before Version 2.41
This module was originally developed at and for Weill Cornell Medical College in Qatar within ITS Advanced Computing Team. With approval from WCMC-Q, this information was generalized and put on github, for which the authors would like to express their gratitude.
Copyright 2016- Jillian Rowe
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install HPC::Runner::Command, copy and paste the appropriate command in to your terminal.
cpanm
cpanm HPC::Runner::Command
CPAN shell
perl -MCPAN -e shell install HPC::Runner::Command
For more information on module installation, please visit the detailed CPAN module installation guide.