sa-learn - train SpamAssassin's Bayesian classifier
sa-learn [options] [file]...
sa-learn [options] --dump [ all | data | magic ]
Options:
--ham Learn messages as ham (non-spam)
--spam Learn messages as spam
--forget Forget a message
--rebuild Rebuild the database if needed
--force-expire Force an expiry run, rebuild every time
--dump [all|data|magic] Display the contents of the Bayes database
Takes optional argument for what to display
--dbpath <path> For dump/import only, specifies (in bayes_path form)
where to read the Bayes DB from
--regexp <re> For dump only, specifies which tokens to
dump based on a regular expression.
-f file, --folders=file Read list of files/directories from file
--dir Ignored; historical compatability
--file Ignored; historical compatability
--mbox Input sources are in mbox format
NOTE: Requires a file, can not use STDIN/pipes
--showdots Show progress using dots
--no-rebuild Skip building databases after scan
-L, --local Operate locally, no network accesses
--import Upgrade data from an earlier database version
-C path, --configpath=path, --config-file=path Path to standard configuration dir
-p prefs, --prefspath=file, --prefs-file=file Set user preferences file
--siteconfigpath=path Path for site configs (def: /etc/mail/spamassassin)
-D, --debug-level Print debugging messages
-V, --version Print version
-h, --help Print usage message
Given a typical selection of your incoming mail classified as spam or ham (non-spam), this tool will feed each mail to SpamAssassin, allowing it to 'learn' what signs are likely to mean spam, and which are likely to mean ham.
Simply run this command once for each of your mail folders, and it will ''learn'' from the mail therein.
Note that globbing in the mail folder names is supported (except when using perl 5.005 in taint-mode); in other words, listing a folder name as * will scan every folder that matches.
SpamAssassin remembers which mail messages it's learnt already, and will not re-learn those messages again, unless you use the --forget option. Messages learnt as spam will have SpamAssassin markup removed, on the fly.
If you make a mistake and scan a mail as ham when it is spam, or vice versa, simply rerun this command with the correct classification, and the mistake will be corrected. SpamAssassin will automatically 'forget' the previous indications.
(Thanks to Michael Bell for this section!)
For a more lengthy description of how this works, go to http://www.paulgraham.com/ and see "A Plan for Spam". It's reasonably readable, even if statistics make me break out in hives.
The short semi-inaccurate version: Given training, a spam heuristics engine can take the most "spammy" and "hammy" words and apply probablistic analysis. Furthermore, once given a basis for the analysis, the engine can continue to learn iteratively by applying both it's non-Bayesian and Bayesian ruleset together to create evolving "intelligence".
SpamAssassin 2.50 and later supports Bayesian spam analysis, in the form of the BAYES rules. This is a new feature, quite powerful, and is disabled until enough messages have been learnt.
The pros of Bayesian spam analysis:
It learns from your mail, so it's tailored to your unique e-mail flow.
And the cons:
i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy to understand. If it generates a false positive or false negative, it's fairly easy to understand why.
With Bayesian analysis, it's all probabilities - "because the past says it's likely as this falls into a probablistic distribution common to past spam in your systems". Tell that to your users! Tell that to the client when he asks "what can I do to change this". (By the way, the answer in this case is "use whitelisting".)
The databases it maintains take quite a lot of resources to store and use.
Still interested? Ok, here's the guidelines for getting this working.
First a high-level overview:
I suggest several thousand of each, placed in SPAM and HAM directories or mailboxes. Yes, you MUST hand-sort this - otherwise the results won't be much better than SpamAssassin on its own. Verify the spamminess/haminess of EVERY message. You're urged to avoid using a publicly available corpus (sample) - this must be taken from YOUR mail server, if it's to be statistically useful. Otherwise, the results may be pretty skewed.
sa-learn --spam /path/to/spam/folder
sa-learn --ham /path/to/ham/folder
...
Let SpamAssassin proceed, learning stuff. When it finds ham and spam it will add the "interesting tokens" to the database.
This can be applied to either ham or spam that has run through the sa-learn processes. It's a bit of a hammer, really, lowering the weighting of the specific tokens in that message (only if that message has been processed before).
cat mailmessage | sa-learn --ham --no-rebuild
This is handy for binding to a key in your mail user agent. It's very fast, as all the time-consuming stuff is deferred until you run with the --rebuild option.
If you don't have a corpus of mail saved to learn, you can let SpamAssassin automatically learn the mail that you receive. If you are autolearning from scratch, the amount of mail you receive will determine how long until the BAYES_* rules are activated.
Learning filters require training to be effective. If you don't train them, they won't work. In addition, you need to train them with new messages regularly to keep them up-to-date, or their data will become stale and impact accuracy.
You need to train with both spam and ham mails. One type of mail alone will not have any effect.
Note that if your mail folders contain things like forwarded spam, discussions of spam-catching rules, etc., this will cause trouble. You should avoid scanning those messages if possible. (An easy way to do this is to move them aside, into a folder which is not scanned.)
If the messages you are learning from have already been filtered through SpamAssassin, the learner will compensate for this. In effect, it learns what each message would look like if you had run spamassassin -d over it in advance.
Another thing to be aware of, is that typically you should aim to train with at least 1000 messages of spam, and 1000 ham messages, if possible. More is better, but anything over about 5000 messages does not improve accuracy significantly in our tests.
Be careful that you train from the same source -- for example, if you train on old spam, but new ham mail, then the classifier will think that a mail with an old date stamp is likely to be spam.
It's also worth noting that training with a very small quantity of ham, will produce atrocious results. You should aim to train with at least the same amount (or more if possible!) of ham data than spam.
On an on-going basis, it's best to keep training the filter to make sure it has fresh data to work from. There are various ways to do this:
This means keeping a copy of all or most of your mail, separated into spam and ham piles, and periodically re-training using those. It produces the best results, but requires more work from you, the user.
(An easy way to do this, by the way, is to create a new folder for 'deleted' messages, and instead of deleting them from other folders, simply move them in there instead. Then keep all spam in a separate folder and never delete it. As long as you remember to move misclassified mails into the correct folder set, it's easy enough to keep up to date.)
Another way to train is to chain the results of the Bayesian classifier back into the training, so it reinforces its own decisions. This is only safe if you then retrain it based on any errors you discover.
SpamAssassin does not support this method, due to experimental results which strongly indicate that it does not work well, and since Bayes is only one part of the resulting score presented to the user (while Bayes may have made the wrong decision about a mail, it may have been overridden by another system).
Also called 'auto-learning' in SpamAssassin. Based on statistical analysis of the SpamAssassin success rates, we can automatically train the Bayesian database with a certain degree of confidence that our training data is accurate.
It should be supplemented with some supervised training in addition, if possible.
This is the default, but can be turned off by setting the SpamAssassin configuration parameter bayes_auto_learn to 0.
This means training on a small number of mails, then only training on messages that SpamAssassin classifies incorrectly. This works, but it takes longer to get it right than a full training session would.
Learn the input message(s) as ham. If you have previously learnt any of the messages as spam, SpamAssassin will forget them first, then re-learn them as ham. Alternatively, if you have previously learnt them as ham, it'll skip them this time around. If the messages have already been filtered through SpamAssassin, the learner will ignore any modifications SpamAssassin may have made.
Learn the input message(s) as spam. If you have previously learnt any of the messages as ham, SpamAssassin will forget them first, then re-learn them as spam. Alternatively, if you have previously learnt them as spam, it'll skip them this time around. If the messages have already been filtered through SpamAssassin, the learner will ignore any modifications SpamAssassin may have made.
Rebuild the databases, typically done after learning with --no-rebuild, or if you wish to periodically clean the Bayes databases once a day on a busy server.
Forces an expiry attempt, regardless of whether it may be necessary or not. Note: This doesn't mean any tokens will actually expire. Please see the EXPIRATION section below.
Forget a given message previously learnt.
Display the contents of the Bayes database. Without an option or with the all option, all magic tokens and data tokens will be displayed. magic will only display magic tokens, and data will only display the data tokens.
Can also use the --dbpath path option to specify the location of the Bayes files to use.
Can also use the --regexp RE option to specify which tokens to display based on a regular expression.
Print help message and exit.
Use the specified path for locating the distributed configuration files. Ignore the default directories (usually /usr/share/spamassassin or similar).
Use the specified path for locating site-specific configuration files. Ignore the default directories (usually /etc/mail/spamassassin or similar).
Read user score preferences from prefs (usually $HOME/.spamassassin/user_prefs).
=item B<-D>, B<--debug-level>
Produce diagnostic output.
Skip the slow rebuilding step which normally takes place after changing database entries. If you plan to scan many folders in a batch, or to learn individual messages one-by-one, it is faster to use this switch and run sa-learn --rebuild once all the folders have been scanned.
Do not perform any network accesses while learning details about the mail messages. This will speed up the learning process, but may result in a slightly lower accuracy.
Note that this is currently ignored, as current versions of SpamAssassin will not perform network access while learning; but future versions may.
If you previously used SpamAssassin's Bayesian learner without the DB_File module installed, it will have created files in other formats, such as GDBM_File, NDBM_File, or SDBM_File. This switch allows you to migrate that old data into the DB_File format. It will overwrite any data currently in the DB_File.
Can also be used with the --dbpath path option to specify the location of the Bayes files to use.
sa-learn and the other parts of SpamAssassin's Bayesian learner, use a set of persistent database files to store the learnt tokens, as follows.
The database of tokens, containing the tokens learnt, their count of occurrences in ham and spam, and the message count of the last message they were seen in.
This database also contains some 'magic' tokens, as follows: the number of ham and spam messages learnt, the number of tokens in the database, the message-count of the last expiry run, the message-count of the oldest token in the database, and the message-count of the current message (to the nearest 5000).
This is a database file, using the first one of the following database modules that SpamAssassin can find in your perl installation: DB_File, GDBM_File, NDBM_File, or SDBM_File.
A map of message-ID to what that message was learnt as. This is used so that SpamAssassin can avoid re-learning a message it's already seen, and so it can reverse the training if you later decide that message was previously learnt incorrectly.
This is a database file, using the first one of the following database modules that SpamAssassin can find in your perl installation: DB_File, GDBM_File, NDBM_File, or SDBM_File.
While SpamAssassin is scanning mails, it needs to track which tokens it uses in its calculations. So that many processes can read the databases simultaneously, but only one can write at a time, this uses a 'journal' file.
When you run sa-learn --rebuild, the journal is read, and the tokens that were accessed during the journal's lifetime will have their last-access time updated in the bayes_toks database.
Since SpamAssassin can auto-learn messages, the Bayes database files could increase perpetually until they fill your disk. To control this, SpamAssassin performs journal synchronization and bayes expiration periodically when certain criteria (listed below) are met.
SpamAssassin can sync the journal and expire the DB tokens either manually or opportunistically. A journal sync is due if --rebuild is passed to sa-learn (manual), or if the following is true (opportunistic):
and either:
or
Expiry is due if --force-expire is passed to sa-learn (manual), or if all of the following are true (opportunistic):
If either the manual or opportunistic method causes an expire run to start, here is the logic that is used:
Go through each of the DB's tokens. Starting at 12hrs, calculate whether or not the token would be expired (based on the difference between the token's atime and the db's newest token atime) and keep the count. Work out from 12hrs exponentially by powers of 2. ie: 12hrs * 1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512 (6144hrs, or 256 days).
The larger the delta, the smaller the number of tokens that will be expired. Conversely, the number of tokens goes up as the delta gets smaller. So starting at the largest atime delta, figure out which delta will expire the most tokens without going above the goal expiration count. Use this to choose the atime delta to use, unless one of the following occurs:
If the expire run gets past this point, it will continue to the end. A new DB is created since the majority of DB libraries don't shrink the DB file when tokens are removed. So we do the "create new, migrate old to new, remove old, rename new" shuffle.
bayes_auto_expire is used to specify whether or not SpamAssassin ought to opportunistically attempt to expire the Bayes databaase. The default is 1 (yes).
bayes_expiry_max_db_size specifies both the auto-expire token count point, as well as the resulting number of tokens after expiry as described above. The default value is 150,000, which is roughly equivalent to a 6Mb database file if you're using DB_File.
bayes_journal_max_size specifies how large the Bayes journal will grow before it is opportunistically synced. The default value is 102400.The sa-learn command is part of the Mail::SpamAssassin Perl module. Install this as a normal Perl module, using perl -MCPAN -e shell, or by hand.
No environment variables, aside from those used by perl, are required to be set.
Mail::SpamAssassin(3) spamassassin(1)
http://www.paulgraham.com/ , Paul Graham's "A Plan For Spam" paper
http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html , Gary Robinson's f(x) and combining algorithms, as used in SpamAssassin
http://www.bgl.nu/~glouis/bogofilter/test6000.html , discussion of various Bayes training regimes, including 'train on error' and unsupervised training
Justin Mason <jm /at/ jmason.org>
