The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
App-ElasticSearch-Utilities-2.7
River stage zero No dependents
7 ++
/ es-search.pl

NAME

es-search.pl - Provides a CLI for quick searches of data in ElasticSearch daily indexes

VERSION

version 2.7

SYNOPSIS

es-search.pl [search string]

Options:

    --help              print help
    --manual            print full manual
    --show              Comma separated list of fields to display, default is ALL, switches to tab output
    --tail              Continue the query until CTRL+C is sent
    --top               Perform a facet on the fields, by a comma separated list of up to 2 items
    --exists            Field which must be present in the document
    --missing           Field which must not be present in the document
    --size              Result size, default is 20
    --asc               Sort by ascending timestamp
    --desc              Sort by descending timestamp (Default)
    --fields            Display the field list for this index!

From App::ElasticSearch::Utilities:

    --local         Use localhost as the elasticsearch host
    --host          ElasticSearch host to connect to
    --port          HTTP port for your cluster
    --noop          Any operations other than GET are disabled
    --timeout       Timeout to ElasticSearch, default 30
    --keep-proxy    Do not remove any proxy settings from %ENV
    --index         Index to run commands against
    --base          For daily indexes, reference only those starting with "logstash"
                     (same as --pattern logstash-* or logstash-DATE)
    --datesep       Date separator, default '.' also (--date-separator)
    --pattern       Use a pattern to operate on the indexes
    --days          If using a pattern or base, how many days back to go, default: all

ARGUMENT GLOBALS

Some options may be specified in the /etc/es-utils.yaml or $HOME/.es-utils.yaml file:

    ---
    host: esproxy.example.com
    port: 80
    timeout: 10

From CLI::Helpers:

    --color             Boolean, enable/disable color, default use git settings
    --verbose           Incremental, increase verbosity
    --debug             Show developer output
    --quiet             Show no output (for cron)

DESCRIPTION

This tool takes a search string parameter to search the cluster. It is in the format of the Lucene query string

Examples might include:

    # Search for past 10 days vhost admin.example.com and client IP 1.2.3.4
    es-search.pl --days=10 --size=100 dst:"admin.example.com" AND src_ip:"1.2.3.4"

    # Search for all apache logs past 5 days with status 500
    es-search.pl program:"apache" AND crit:500

    # Search for all apache logs past 5 days with status 500 show only file and out_bytes
    es-search.pl program:"apache" AND crit:500 --show file,out_bytes

    # Search for ip subnet client IP 1.2.3.0 to 1.2.3.255 or 1.2.0.0 to 1.2.255.255
    es-search.pl --size=100 dst:"admin.example.com" AND src_ip:"1.2.3.*"
    es-search.pl --size=100 dst:"admin.example.com" AND src_ip:"1.2.*"

    # Show the top src_ip for 'www.example.com'
    es-search.pl --base access dst:www.example.com --top src_ip

    # Tail the access log for www.example.com 404's
    es-search.pl --base access --tail --show src_ip,file,referer_domain dst:www.example.com AND crit:404

Helpful in building queries is the --fields options which lists the fields:

    es-search.pl --fields

NAME

es-search.pl - Search a logging cluster for information

OPTIONS

help

Print this message and exit

manual

Print detailed help with examples

show

Comma separated list of fields to display in the dump of the data

    --show src_ip,crit,file,out_bytes
tail

Repeats the query every second until CTRL+C is hit, displaying new results. Due to the implementation, this mode enforces that only the most recent indices are searched. Also, given the output is continuous, you must specify --show with this option.

top

Comma separated list of fields to facet on. Given that this uses scripted facets for multi-field facets, it is limited to faceting on up to 2 fields. This option is not available when using --tail

    --top src_ip
exists

Filter results to those containing a valid, not null field

    --exists referer

Only show records with a referer field in the document.

missing

Filter results to those not containing a valid, not null field

    --missing referer

Only show records without a referer field in the document.

fields

Display a list of searchable fields

index

Search only this index for data, may also be a comma separated list

days

The number of days back to search, the default is 5

base

Index base name, will be expanded using the days back parameter. The default is 'logstash' which will expand to 'logstash-YYYY.MM.DD'

size

The number of results to show, default is 20.

AUTHOR

Brad Lhotsky <brad@divisionbyzero.net>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2012 by Brad Lhotsky.

This is free software, licensed under:

  The (three-clause) BSD License