VBTK - Virtual Brent Toolkit - A generic toolkit for system monitoring
VBTK is a collection of modules which can be used to build a complex monitoring system completely written in perl. It's system monitoring abilities are, at the moment, mostly limited to solaris unix servers. But it also supports monitoring through SNMP, HTTP, SMTP, POP3, etc., and the framework is easily expanded. More modules will be added over time. See the sections below on File System Layout, Environment Variables, Server and Control Modules, and Data-Gathering Modules.
Note that the VB toolkit is just a group of modules. You write the final perl program which runs them yourself. But don't let that deter you, everything is defaulted and quite simple, so the final perl program is usually very simple. Also, there are many examples and templates to copy from.
The following modules are used in the VBTK collection to define the VB Server, and control the various monitoring processes.
This module allows the definition of a VB Server daemon process. This process is responsible for gathering, monitoring, and displaying the statuses and associated text of of all test objects. It has an embedded web server which is the access point for client processes to submit statuses, as well as for users to view statuses. See VBTK::Server for more details.
This module allows the definition of actions which can be triggered by the VB server based on status changes.
There are also sub-classes of the VBTK::Actions module. These provide convenient defaults for two common types of actions - email and paging. See VBTK::Actions for more details.
This module is used to start/stop all the various client and server processes which make up the VBTK monitoring system. See VBTK::Controller for more details.
The following modules are run as part of client processes which gather data, process it, and report it to the VB server. Each module is setup with common defaults so that it can be used with minimal configuration. Some modules have sub-classes which provide additional groups of common defaults, to further simplify configuration. See each module's man page for more details.
This module is the heart of most of the monitoring scripts. It is a simple command-line-execution wrapper. It runs a command on the command line and then parses the output. It can also tail the output of a log file. If multiple VBTK::Wrapper objects are defined in the same perl process, the VBTK engine will time-share between them. It passes it's collected data on to the VBTK::Parser module.
There are also sub-classes of the VBTK::Wrapper module. These provide convenient defaults for various monitoring configurations, including running vmstat, df, metatstat (DiskSuite), vxprint (VXVM), ping, etc. (Mostly Solaris utilities) See VBTK::Wrapper for more details.
This module allows the monitoring of systems through Snmp. It retrieves a specified list of SNMP oid's on a specified interval and passes them on to the VBTK::Parser module.
There are also sub-classes of the VBTK::Snmp module. These provide convenient defaults for various monitoring configurations, including monitoring network I/O, Dynamo, NT cpu utilization, etc. See VBTK::Snmp for more details.
This module allows the monitoring of a web server. It retrieves HTML from specified URL's on a specified interval. It can establish an HTML baseline and compare the retrieved HTML to the original baseline with each pass. It also records the response time of the HTML retrieval. See VBTK::Http for more details.
The module allows the monitoring of a TCP listener. It simply attempts to connect to a specified host and port and then logs the success or failure. It's good for monitoring any TCP listener. It records the reponse time of the connection attempt. See VBTK::Tcp for more details.
This module allows the monitoring of databases through the use of the DBI module. It executes pre-defined SQL on a specified interval, and passes the results on to the VBTK::Parser module.
There are also sub-classes of the VBTK::DBI module. These provide convenient defaults for various monitoring configurations, including monitoring oracle tablespaces, io rates, and blocking processes.
This module allows the monitoring of an SMTP server. It sends an email through the specified SMTP server on a specified interval, measures the response time, and passes the result on to the VBTK::Parser module.
This module allows the monitoring of a POP3 server. It connects to the POP3 server on a specified interval, measures the response time, and passes the result on to the VBTK::Parser module. It can also be configured to retrieve email with a specified Subject, and measure and store the round-trip time. This is usually used together with the VBTK::Smtp module to test the full mail cycle.
This module allows the monitoring of ascii text logs. It will tail the specified log or logs, searching for patterns, and setting VBObject statuses accordingly.
This module is never called directly from a monitoring script. It is the engine responsible for parsing through the data retrieved by the data-gathering modules and filtering, ignoring, or re-arranging the data as specified. It makes the decisions about what status to set, and handles contacting the VB server to report the status, update graphs, store text, etc. See VBTK::Parser for more details.
The VB toolkit can make use of several environment variables to override the defaults entered when VBTK was first installed. If you go with the defaults, it makes things much simpler.
This is the root directory of the VBTK files. (Defaults to /usr/vbtk).
The TCP port on which the VBServer web server will be available and which the client processes will report their statuses into the VBServer. (Defaults to 4712).
The URI which the client processes will use to contact the VBServer when reporting their statuses. (Defaults to 'http://vbserver:$VBPORT') Note that if you just setup a DNS alias for 'vbserver', then you don't have to change the default.
The path to a file where passwords can be stored for automated access by client processes. This is currently only used by the VBTK::DBI module.
The directory where the logs of STDOUT from each of the VBServer and client processes will be stored. (Defaults to '$VBHOME/logs')
The directory where the object database will be stored. Don't put anything else into this directory, it should just be for the object database. (Defaults to '$VBHOME/vbobj') You can override this in the call to new VBTK::Server with the ObjectDir parameter if you want.
The location of the rrdtool binary. Defaults to '/usr/local/bin/rrdtool'.
By default, the VB toolkit installs into a single directory as designated by the environment variable VBHOME. Sub-directories are then setup as follows. See the section on Environment Variables for details on how to relocate these from their default settings.
Location of some administrative scripts used for installing and replicating the VBTK software.
Location of user-defined configuration scripts
A directory containing example files which can be copied into the 'conf' directory and then customized.
Location of STDOUT log files from the VBServer and client processes.
Directory in which MIB files can be installed for use by the VBTK::Snmp module.
Directory in which performance related log files are usually stored by default. These log files contain the data gathered by the client monitoring processes. Their format and location is determined by the 'LogFile', 'LogHeader', and 'LogDetail' parameters passed in the contructors of the data-gathering modules.
Directory in which can be placed shell or other scripts used by the data-gathering processes to collect their data.
Directory in which the VBServer stores it's VB Object database. For each VB object created, the VBServer will create a sub-directory using the objects full name. All history data, RRD databases, meta data, etc will be stored in this sub-directory.
Document root directory for the VBServer's built-in web server. The '.phtml' and images files in this directory are all user-customizable, so feel free to customize the web interface as you like. See the section on Customizing the Web Interface in the VBTK::Server documentation for more details.
The VB toolkit depends on several other perl modules, which are readily available from CPAN.
The following modules are required. They are available from CPAN - http://www.cpan.org/modules.
The simplest way to install these is to just use 'perl -MCPAN -e shell'. Once in the shell, just type 'install Date::Manip', 'install Storable', etc.
In addition, the VB toolkit can make use of the following modules if installed.
Allows for creation of RRD databases and dynamic generation of graphs if the rrdtool is installed along with it's dependencies. You can find the source at http://www.rrdtool.com/index.html . I strongly recommend installing this!
Allows for SNMP monitoring if the SNMP module is installed along with it's dependencies. This requires requires the Net-SNMP toolkit library available from http://sourceforge.net/projects/net-snmp .
Allows for database monitoring via the DBI module, if the DBI module is installed along with it's dependencies.
Follow the README instructions for doing the usual Perl module 'make', 'make test' and 'make install'. Once that's done, getting a vanilla system up and running is fairly easy.
Before we start configuring anything, you need to plan your monitoring architecture. Draw a map of this on paper first, taking into consideration the following areas. Just start with something simple at first.
Decide where you'll run the master VBServer process. This is the instance where you'll edit the config files for all other hosts and from which those config files will be replicated when you use the 'vbc sync' command.
Note that the VBServer process can become quite intensive if you have a lot of objects to monitor, so choose a machine which you can dedicate to just monitoring. Also, make sure it's a stable machine (I guess that rules out Windows...) which isn't going to get un-plugged by the janitor or crash regularly.
List out your hosts and what services they run. Decide what you'll want to monitor. (ie: disk space, cpu utilization, web server, ftp server, mail server, etc.)
Some monitoring is done locally, such as disk space, cpu utilization, disk volume status, log tailing, etc. Other monitoring is usually done remotely, such as ping and testing of smtp, snmp, pop3, and other TCP services. It's usually easiest to run all the remote tests from one or two machines. Also you may want to run the same test from multiple locations. For example, you might want to test your HTTP server from both inside and outside the network. That way you could better differentiate between a network failure and a HTTP server failure.
If you have hosts you want to monitor which are not on the same LAN as the master VBServer, it's best to setup a slave VBServer on the remote LAN. The clients send a lot of data to their VBServer and you don't want this all going over the WAN. In addition, the master VBServer and slave VBServer can be configured to heartbeat each other, so that in case one dies, the other can warn you.
You can access the web interface from the Master node or from any Slave node. They will all show all the objects in one interface. But don't expose this interface to the public internet, since I'm not sure about vulnerabilities.
Designate one host as the place to run your master VBServer. This will be where you make changes to the config files. You can then use the 'vbc sync' command on each of the other hosts to download the changes.
Note that you don't have to use the 'sync' functionality. You could just use rsync or something similar to copy the config files around, or you could just edit the config files independently on each host. I prefer to have one master and sync out from there, because it makes it easier to keep track of all the config differences.
Follow the 'PER-SERVER CONFIGURATION' instructions below for just the VBServer host. When you run the installer, answer (M)aster when prompted for which type of processes you'll be running on this host.
Look in the $VBHOME/examples directory and you'll find example config files. These are actually perl programs, so it will help if you know some perl. The 'vbc' config file will have already been copied over to the $VBHOME/conf directory. Open up the 'vbc' config file. This is the controller file which defines which instances will be running on each host. You should already see the master VBServer host listed there. Add in all your other hosts, and specify any slave servers.
Copy the 'vbserver' config file from the 'examples' directory into the 'conf' directory and edit it. Just read the comments.
Run 'conf/vbc start vbserver' and then check the log file 'logs/vbserver...' for errors. Now try to connect with a web browser to 'http://<hostname>:4712'.
Now that you've drawn your monitoring map, let's get the various servers setup. The software install is identical regardless of what you're going to run on the host. If you have hosts which won't be running any local tests, then you obviously don't need to install anything there.
If you're running this on a unix machine, you don't want to run it as superuser, so create a user 'vb' and group 'vb' to run all the processes under.
useradd vbtk groupadd vbtk
Create a home directory for the VBTK software. I suggest '/usr/vbtk', which is the default. Change this directory to be owned by the 'vbtk' user and to be the 'vbtk' user's home directory.
mkdir /usr/vbtk; chown vb:vb /usr/vbtk
If you're going to use a directory other than '/usr/vbtk', then make sure you set the environment variable $VBHOME to the new location. Make sure this is always set before you run any scripts.
This will check the VBHOME directory and prompt you for either the location of the VBTK install directory or a VBServerURI from which it can sync config files. It also does some setup in the $VBHOME directory.
perl -MVBTK -e install
Once you have the initial VBServer running, read through the config files in the examples directory and copy ones you want to use over to the conf directory. Remember you have to list these config files in the 'vbc' config file in order to start them with the 'vbc' command.
You can also look through the various modules in the Data Gathering Modules section of this document for help on creating new config files.
Don't expose the web interface to the public internet! I make no claims about how secure it is. There may be vulnerabilities which I don't know about.
Before you can delete an object, make sure that there isn't a client process which is still sending sending a status for that object. Otherwise it will just get recreated with the next status submission. If you're not sure which client process is setting the status, look under the 'Info' tab for that object for the 'Script Name' and 'Running From' values. This shows you which script is setting the status. Once you think you've fixed the script, check the object history to see if it's still being updated. If not, go to the 'vbobj' directory, on the Master or Slave server to which this object is reported, and delete the sub-directory which matches the object name. After about a minute, the object will disappear from the web interface.
If you've changed the number of entries being passed in the RrdColumns entry, or if you've changed the CF specification, then you probably need to rebuild the corresponding RRD database. Look in the VBHOME/vbobj directory for a directory named the same as the object and delete the RRD.db file. Then restart the client process which monitors the object and the RRD.db file will be regenerated.
The VBTK toolkit is written such that additional data-gathering modules can be added on to it fairly easily. See the 'examples' sub directory under the VBTK home directory for examples of how to write your own data-gathering modules, or look at the code of the data-gathering modules listed above.
The @delta array used in the 'VBDetail' and 'LogDetail' parms of data-gathering arrays doesn't yet take into account 32-bit and 64-bit roll-over. So you may get an occasional very large negative number when counters roll-over. This will be fixed in a future release.
I did some initial testing running this under Windows, but had too many problems with the 'fork' emulation. It just didn't work quite right. Also, windows doesn't support 'alarm', which I use quite a bit. So it's not going to work. It runs great on Cygwin however, so this is probably the best way to go if you want to run it on a windows platform.
The VBServer engine uses more resources than I would have liked, but seems to run okay. On my Sun Netra T1 440MHz server, with about 100 objects reporting in, it runs at about 3-5% all the time. I'll work on getting that down lower in a future release.
The VBServer process will default to GMT if Date::Manip can't determine the timezone. See "Timezones" in Date::Manip for details on how to set the Timezone.
I have several things in mind, but feel free to send suggestions.
There needs to be a single place to go to see a log of all the actions fired recently. In addtion, I would like to add the option of requiring actions to be acknowledged. That way if your sysadmin misses the first page, it will keep paging every n minutes, until they log in and acknowledge the problem.
I'd like to add the option of saving the cookie database somewhere on disk, so that it doesn't get wiped out every time you restart.
I'd like to add an 'Admin' page in the web interface to allow administrative actions such as deleting an object, resetting the RRD database, etc.
I'd like to add a 'MaxRepeat' option for object history.
And a special thanks to all the great perl authors who's libraries I used to build this. I couldn't have done it without you! CPAN is great!
Brent Henry, vbtoolkit@yahoo.com
Copyright (C) 1996-2002 Brent Henry
This program is free software; you can redistribute it and/or modify it under the terms of version 2 of the GNU General Public License as published by the Free Software Foundation available at: http://www.gnu.org/copyleft/gpl.html
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
To install VBTK, copy and paste the appropriate command in to your terminal.
cpanm
cpanm VBTK
CPAN shell
perl -MCPAN -e shell install VBTK
For more information on module installation, please visit the detailed CPAN module installation guide.