The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
RTG-Report-1.17
River stage zero No dependents
++
/ record_consolidator.pl

NAME

record_consolidator.pl - consolidate old RTG data to reduce database size.

VERSION

1.0.0

SYNOPSIS

  record_consolidator

Options for controlling consolidation are found in rtgreport.conf.

DESCRIPTION

record_consolidator.pl - Consolidates RTG data into summary records. We monitor tens of thousands of network ports and have hundreds of gigabytes of RTG data. When disk space or performance becomes an issue, the RTG supplied tool is a pruning script that simply deletes old data. Rather than deleting that data, I wanted to condense it, preserving the essence of the data. I borrowed a concept from Tobias Otiker and his RRD databases--rather than discard old data, reduce its granularity.

In testing, I discovered that 15 minute averages still work perfectly with unaltered versions of the RTG CGI application. Any more than 15 minute averages and graphs stop rendering. Dropping from 5 to 15 minute averages reduces the disk space required by old data by 66%.

How you use the consolidator script will be dictated by your business needs. I suggest using the consolidator script with two sets of settings. Old data that is no longer needed for graph publication should be pruned to 60 minute averages. A setting of 190/60 will do this. Then have another tier of pruning where you consolidate data older than 2-3 months to 15 minute averages. A setting of 90/15 would do that.

CONFIGURATION

The consolidator script is controlled by two settings in the rtgreport.conf config file:

 [Pruning]
  days     = 190
  interval = 60

Days controls what data you want to alter. A setting of 190 as shown will only condense data older than 190 days. Interval controls what interval the consolidated records will be. A setting of 15 will condense the data to 15 minute averages (66% reduction). A setting of 60 as shown will reduce the data to 60 minute averages for a 91% reduction in disk space.

DEPENDENCIES

 Config::Std
 Date::Calc
 Date::Format
 DBIx::Simple

AUTHOR

Matt Simerson <msimerson@cpan.org>

CHANGES

LICENSE AND COPYRIGHT

Copyright (c) 2008 Layered Technologies, Inc. All rights reserved.

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.