The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Log-Unrotate-1.33
River stage one • 1 direct dependent • 1 total dependent
1 ++
/ Log::Unrotate

NAME

Log::Unrotate - Incremental log reader with a transparent rotation handling

VERSION

version 1.33

SYNOPSIS

  use Log::Unrotate;

  my $reader = Log::Unrotate->new({
      log => 'xxx.log',
      pos => 'xxx.pos',
  });

  my $line = $reader->read();
  my $another_line = $reader->read();

  $reader->commit(); # serialize the position on disk into 'pos' file

  my $position = $reader->position();
  $reader->read();
  $reader->read();
  $reader->commit($position); # rollback the last 2 reads

  my $lag = $reader->lag();

DESCRIPTION

Log::Unrotate allows you to read any log file incrementally and transparently.

Incrementally means that you can store store the reading position to the special file ("pos-file") using commit(), restart the process, and then continue from where you left.

Transparently means that Log::Unrotate automatically jumps from one log to the next. For example, if you were reading foo.log, then stored the position and left for a day, and then while you were away, foo.log got renamed to foo.log.1, while the new foo.log got some new content in it, Log::Unrotate will find the right log and give you the remaining lines from foo.log.1 before moving to foo.log.

Even better, it will do the right thing even if the log rotation happens while you were reading the log.

Log::Unrotate tries really hard to never skip any data from logs. If it's not sure about what to do, it throws an exception. This is an extremely rare situation, and it is a good default for building a simple and robust message queue on top of this class, but if you prefer a quick-and-dirty recovering, you can enable autofix_cursor option.

METHODS

new($params)

Creates a new unrotate object.

pos

Name of a file to store log reading position. Will be created automatically if missing.

Value '-' means not to use a position file. I.e., pretend it doesn't exist at the start and ignore commit calls.

cursor

Instead of pos file, you can specify any custom cursor. See Log::Unrotate::Cursor for the cursor API details.

autofix_cursor

Recreate a cursor if it's broken.

Warning will be printed on recovery.

rollback_period

Time period in seconds. If rollback_period is greater than 0, commit method will save some positions history (at least one previous position older then rollback_period would be preserved) to allow recovery when the last position is broken for some reason. (Position may sometimes become invalid because of the host's hard reboot.)

The feature is enabled by default (with value 300), set to 0 to disable, or set to some greater value if your heavily-loaded host is not flushing its filesystem buffers on disk this often.

log

Name of a log file. Value - means standard input stream.

start

Describes the initialization behavior of new cursors. Allowed values: begin (default), end, first.

  • When start is begin, we'll read current log from the beginning.

  • When start is end, we'll put current position in log at the end (useful for big files when some new script don't need to read everything).

  • When start is first, Log::Unrotate will start from the oldest log file available.

I.e., if there are foo.log, foo.log.1, and foo.log.2, begin will start from the top of foo.log, end will skip to the bottom of foo.log, while first will start from the top of foo.log.2.

end

Describes the reading behavior when we reach the end of a log. Allowed values: fixed (default), future.

  • When end is fixed, the log is read up to the position where it ended when the reader object was created. This is the default, so you don't wait in a reading loop indefinitely because somebody keeps adding new lines to the log.

  • When end is future, it allows the reading of the part of the log that was appended after the reader was created (useful for reading from stdin).

lock

Describes the locking behaviour. Allowed values: none (default), blocking, nonblocking.

  • When lock is blocking, lock named pos.lock will be acquired in the blocking mode.

  • When lock is nonblocking, lock named pos.lock will be acquired in the nonblocking mode; if lock file is already locked, exception will be raised.

check_lastline

This flag is set by default. It enables content checks when detecting log rotations. There is actually no reason to disable this option.

check_inode

Enable inode checks when detecting log rotations. This option should not be enabled when retrieving logs via rsync or some other way which modifies inodes.

This flag is disabled by default, because check_lastline is superior and should be enough for finding the right file.

read()

Read a line from the log file.

position()

Get your current position in log as an object passible to commit().

commit()
commit($position)

Save the current position to the pos-file. You can also save some other position, previosly obtained with position().

Pos-file gets commited using a temporary file, so it won't be lost if disk space is depleted.

lag()

Get the size of data remaining to be read, in bytes.

It takes all log files into account, so if you're in the middle of foo.log.1, it will return the size of remaining data in it, plus the size of foo.log (if it exists).

log_number()

Get the current log's number.

log_name()

Get the log's name. Doesn't contain .N postfix even if cursor points to old log file.

BUGS & CAVEATS

To find and open correct log is a race-condition-prone task.

This module was used in production environment for many years, and many bugs were found and fixed. The only known case when position file can become broken is when logrotate is invoked twice in *very* short amount of time, which should never be a case.

Don't set the check_inode option on virtual hosts, especially on openvz-based ones. If you move your data, inodes of files will change and your position file will become broken. In fact, don't set check_inode at all, it's deprecated.

The logrotate config should not use the compress option to make that module function properly. If you need to compress logs, set delaycompress option too.

This module expects the logs to be named foo.log, foo.log.1, foo.log.2, etc. Skipping some numbers in the sequence is ok, but postfixes should be *positive integers* to be properly sorted. If you use some other naming scheme, for example, foo.log.20130604-140500, you're out of luck. Patches welcome!

AUTHORS

Andrei Mishchenko druxa@yandex-team.ru, Vyacheslav Matjukhin me@berekuk.ru.

SEE ALSO

File::LogReader - another implementation of the same idea.

unrotate - console script for reading logs.

COPYRIGHT

Copyright (c) 2006-2013 Yandex LTD. All rights reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

See <http://www.perl.com/perl/misc/Artistic.html>