Stem::LogTail Design Notes

The Stem::LogTail module provides a service similar to the standard Unix
program tail -f. It constructs a Cell that can watch a file for changes
in it size or inode and then sends the new data to a destination. The
destination is a Logical Log created elsewhere in the Stem application
and it can be located on any Hub on the network. Also file status
changes such as not being found, first time it is opened, it was
truncated, etc. can be sent to a different Logical Log than the data.
This module is a critical part of the StemLog application which
transfers active log files from one system to another.

The action of checking the file is triggered by a call to the tail_cmd
method. There are two main ways of triggering it, either by an internal
timer or via a command message directed at this Cell. The timer is
configured when the Cell is created and its resolution in in seconds. So
e.g. you can cause a file check to occur every 15 seconds. If a 'tail'
command message is sent to this Cell, it will also trigger a file
check. The command message is usually sent from a Stem::Cron entry
configured elsewhere. In either case new data is sent to the 'data_log'
Log and status is sent to the 'status_log' Log.

The primary configuration parameter is the path to the file to be
checked. Also the required 'data_log', optional 'status_log' and
'repeat_interval' are set in the configuration.

When the file is checked, the current size and inode are compared to the
previous values. If either has changed, then either data and/or status log
entries are sent.

NOTE: The modification time of a file is not currently checked for
changes. The reason is that there is no way to determine if a file
changed (or what data has changed) if the file just has its modification
time changed (by touch or utime). This means that there is a weakness in
just checking for the file size changing. If a file were to be truncated
and then written to with the same amount of data as it had when it was
last checked, a tail operation would not report any new data. This bug
exists in both the GNU and Solaris tail programs. The only way around
this is to check for modification time changes and trust
this. Stem::LogTail will support an option for doing this instead of
checking file size but it is up to the user to assure that the file will
only change its modification time if its data actually changes. Most
logs generated by programs have this attribute.