NAME

Hadoop::HDFS::Command

VERSION

version 0.002

SYNOPSIS

    use Hadoop::HDFS::Command;
    my $hdfs = Hadoop::HDFS::Command->new;
    my @rv = $hdfs->$command( @command_args );

DESCRIPTION

This is a simple wrapper around the hdfs commandline to make them easier to call from Perl and parse their output.

The interface is partially done at the moment (see the implemented wrappers down below).

You can always use the WebHDFS to do similar operations instead of failling back to the commandline. However there are several benefits of using the cli; i) you'll end up with a single JVM invocation, so the response might be faster ii) Some functionality / endpoints might be buggy for WebHDFS but might work with the cli (for example escaping certain values is broken in some versions, but works with the cli).

NAME

Hadoop::HDFS::Command - Wrappers for various hadoop hdfs cli commands

METHODS

new

The constructor. Available attributes are listed below.

cmd_hdfs

Default value is /usr/bin/hdfs. This option needs to be altered if you have the `hdfs` command in some other place.

enable_log :Bool

Can be used to enable the internal logging feature. Disabled by default.

dfs

One of the top level commands, including an interface to the sub-commands listed below. The calling convention of the sub commands is as simple as:

    my @rv = $hdfs->dfs( \%options, $sub_command => @subcommand_args );
    # options hash is optional
    my @rv = $hdfs->dfs( $sub_command => @subcommand_args );

Available options are listed below:

ignore_fail :Bool

Global.

silent :Bool

Global.

want_epoch :Bool

Only used for ls. Converts timestamps to epoch.

callback :CODE

Only used for ls. The callback always needs to return true to continue processing, returning false from it will short-circuit the processor.

du

The @subcommand_args can have these defined: -s, -h.

    my @rv = $hdfs->dfs( du => @subcommand_args => $hdfs_path );
    my @rv = $hdfs->dfs( du => qw( -h -s ) => "/tmp" );
    my @rv = $hdfs->dfs(
                {   
                    ignore_fail => 1,
                    silent      => 1,
                },  
                du => -s => @hdfs_paths,
            );

ls

The @subcommand_args can have these defined: -d, -h, R.

    my @rv = $hdfs->dfs( ls => @subcommand_args => $hdfs_path );

The callback can be used to prevent buffering and process the result set yourself. The callback always needs to return true to continue processing. If you want to skip some entries but continue processing then a true value needs to be returned. A bare return (which is false) will short circuit the iterator and discard any remaining records.

    my %options = (
        callback => sub {
            # This callback will receive a hash meta-data about the file.
            my $file = shift;
            if ( $file->{type} eq 'dir' ) {
                # do something
            }
            
            # skip this one, but continue processing
            return 1 if $file->{type} ne 'file';
            
            # do something
            
            return if $something_really_bad_so_end_this_processor;
            
            # continue processing
            return 1;
        },
        # The meta-data passed to the callback will have an "epoch"
        # key set when this is true.
        want_epoch => 1,
    );
    # execute the command recursively on the path
    $hdfs->dfs( \%options, ls => -R => $hdfs_path );

mv

    my @rv = $hdfs->dfs( mv => $hdfs_source_path, $hdfs_dest_path );

put

The @subcommand_args can have these defined: -f, -p, -l

    $hdfs->dfs( put => @subcommand_args, $local_path, $hdfs_path );
    # notice the additional "-"
    $hdfs->dfs( put => '-f', '-', $hdfs_path, $in_memory_data );

rm

The @subcommand_args can have these defined: -f, -r, -skipTrash

    $hdfs->dfs( rm => @subcommand_args, $hdfs_path );

SEE ALSO

`hdfs dfs -help`.

AUTHOR

Burak Gursoy <burak@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2016 by Burak Gursoy.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

cpanm Hadoop::HDFS::Command

perl -MCPAN -e shell
install Hadoop::HDFS::Command