Sendmail::M4::Mail8 - Stop fake MX and most spammers, sendmail M4 hack file
Version 0.33 (Beta)
Now running at mail.celmorlauren.com our own mail server, and has been doing so since 0.23
SPAM consitutes the bulk of e-mail on the internet, many methods exist to fight this scurge, some better than others. However we think that this module is the simplest, quickest and most efective, relying as it does on the basic power of sendmail macros for most of its methods.
Local system, local private IP address, contained in w, {VirtHost}, R, mail1.db and mail2.db.
Generally exempted from tests, except for From: and Reply-to:
External system, that fails one or more tests, and you want to receive mail from, these must be contained by mail4.db
Generally exempted from tests, except for Received:
External systems that are not allowed to send e-mail.
Currently requires additional Perl helpers that are not included in the CPAN distribution.
For basic protection use the standard sendmail access database.
Only {GoodRely} may have a f domain-name part that is local to this system.
{Bounce} f of <>, only permitted on {Paranoid} values of 3 or less.
All hosts s|(HELO), apart from {GoodRelay} and {BadRelay}, must
Not pretend to be one of our domains or a local private IP address.
Must either be the same as {client_name} or resolve to be the same as {client_addr}.
f|(FROM:) domains names containing yahoo or hotmail may only come from MX systems containing these names, also the Reply-to: must either not exist or not differ.
Not contain more than 2 parts of their {client_addr} IP address encoded within their domain name, or be overly numeric.
This traps the bulk of zombie spammers.
Traps pessimists like hotmail as well, as they use very numeric MX mail relays. As I do not know of anyone who uses hotmail, nothing this end will be done about it. It is upto <hotmail> to have confidence in the mail they are relaying, and hence use clearer MX domain names.
{Bounce} delt with depending on {Paranoid}, many are callback verify requests an unhelpfull sudo bounce that some other anti-spam systems have adopted.
A level of 3 says OK to anything (even domains not hosted here)! Systems that waste our time with dubious rubbish like callback verify deserve to get rubbish back.
mail3 trouble ticket, one shot e-mail allowed from web-site will be delt with here.
Only with {Paranoid} of 0 will any {Bounce} with data be accepted!
Must be the same as b<f>, much SPAM uses a From: of yahoo or hotmail or some other mail address in a attempt to evade the local_check_mail tests. Also as only {BadRelay} is exempted it traps your own users who are trying to send out as someone else, or perhaps has become a zombie.
Not permitted for yahoo or hotmail domains, otherwise must be the same domain as f
Detection of fake headers, spammers often included faked headers perporting to come from one of your own domains.
Web-mail is so insecure and open to abuse by spammers, that with a {Paranoid} setting of 2 and above it will be refused.
As all systems have an IP address and most have some sought of domain-name, it is possible to base the protection on wether the IP ties up with whom they claim to be at the <helo> stage. You can set sendmail to be picky about this. But many peoples IP address does not resolve to what they would like, its easy to setup domain to IP via people like network solutions, but the other way round needs a friendy ISP. So as this is a common problem, base the protection on the helo resolving to their IP.
Next check that their domain does not contain their IP encoded somehow, people who are not real MXs tend to have numeric user addresses, this has tuning to control how strict this is.
Keep a record of whom the system has sent mail to, so that we have a chance of spotting spammers using fake bounces to fill up a users email box, at the most paranoid this refuses all bounces. That causes some problems with some systems who use fake bounces to check wether you are an MX, some even come from a completly different domain to the one being talked to at the time!??! Stupid or what?
Next check that the From address is not pretending to be one of your own hosted domains, ie the IP is external and is not known to you as an outside user.
After that noraml sendmail DB files will do the rest, use the cookbook, all you need to know is there.
Sendmail::M4::Utils does most of the work for this module, all this does is format the rules and supply a default name for the hack. Various tuning methods exist, and of course you can add your own rules to this.
This module is non OO, and exports the methods descriped under EXPORTS.
Ian McNulty, celmorlauren limited (registered in England & Wales 5418604).
email <development@celmorlauren.com>
Sendmail::M4::Utils the module created to make testing this easier
This configures this module, and is allways required first.
Expected/Allowed values allways as a (hash value pairing), see Sendmail::M4::Utils for hash=>value pairings it expects, the list bellow are either default values or additional for use by this.
Sendmail::M4::Utils
file SCALAR with default value of "mail8-stop-fake-mx.m4", build SCALAR with default value of 1 install SCALAR with default value of 1 test SCALAR with default value of 1 paranoid SCALAR see heading below for values
not paranoid at all, has local users and is content to accept bounces and "callback verify" sudo bounces.
Standard sendmail rules and databases will handle user and bounce requests. This just verifys that the sending host appears to be legimate. Assuming that the hit rate on the system is not too great, use sendmail "milters" as well to take care of "spam" and "viruses"
slightly paranoid, has local users and is content to accept "callback verify" sudo bounces, but will refuse any bounce request that really is a bounce, that is a bounce with data.
mildly paranoid, is a relay host with no local users, will say OK to all "callback verify" requests that refer to hosted domains, regardless of wether the user exits or not! Refuses all real bounces.
paranoid, is a hassled relay host, will just say OK to any "callback verify" request, regardless of wether it relays for that domain or not! Refuses all real bounces.
fairly paranoid, is a really hasseled relay host, and has no time for any type of bounce, all refused. Most e-mail and even more bounces are bogus.
copyright message to list at the start of the hack, anything supplied will replace the first two lines below.
Copyright (c) 2007 celmorlauren Limited England Author: Ian McNulty <development\@celmorlauren.com>
this should live in /usr/share/sendmail/hack/
some settings that are advised FEATURE(`access_db', `hash -T<TMPF> -o /etc/mail/access.db') FEATURE(`greet_pause', `2000') define(`confPRIVACY_FLAGS', `goaway')
This is really a reminder to use VERSIONID with your own value, or just use this to use the default
VERSIONID "ANTI SPAM & FAKE MX"
Required statement, this inserts required statements into the hack file.
This inserts required statements before and after LOCAL_CONFIG, you may add more statements that belong here.
Main items "-" added to confOPERATORS KRlookup for DNS check on HELO host name H*: $>+ScreenHeader to check received headers KMath arith to join the IP address together into a single token KCleanFrom regex to enable checking of Header line "From:"
KZombie program /etc/mail/mail8/mail8_zombie this is included in the script regardless of wether it is installed or not. Included as part of the distro, install it to get the full benifits. dnl white list no other way to to let these past Kmail4db hash -o -a.FOUND /etc/mail/mail8/mail4.db Standard black list, checked at Reply-to: Header dnl standard black list Kstdaccessdb hash -o -a.FOUND /etc/mail/access.db
This enables the use of the additional Perl scripts to identify and block bogus e-mail hosts, especialy when the site is being bombed by an abusive system.
None of the scripts are currently available on CPAN, and there is no current intention of releasing them at this time, this is mostly due to the extra system setup required, such as interfaces to the iptables firewall script bring used!
If you would like to use these, contact celmorlauren for help.
These are configured automatically by the above PerlHelpers, and are only useable with PerlHelpers, if the above has not been defined then this returns without doing anything.
However there is one exception to this, mail4 is always required as there is no other way of allowing "broken mail systems" that you want to accept mail from!
To keep sendmail error messages down (/var/log/mail) ensure you create all the required database's by hand, mail4 at the very least!
For manual creation of files, use vi ###; makemap hash ###.db <### where ### is the database source.
/etc/mail/mail8/mail9.db ip (address port 25) to TarPit in firewall rules /etc/mail/mail8/mail8.db refuse connect to SPAMMER access.db also does this and more /etc/mail/mail8/mail4.db allow, OK this host would fail tests /etc/mail/mail8/mail3.db single shot, allow, like mail4 /etc/mail/mail8/mail2.db relays hosted domains $=R, $=w, & ${VirtHost} also does this /etc/mail/mail8/mail1.db relays internal hosts by IP 192.168.#.# assummed local 172.16.#.# assummed local 10.#.#.# assummed local
NOTE This files are all optional, so this can be specified even if none of these exist.
The single useable argument if SCALAR will place the DataBases in /var/tmp/mail8, which enables you to test with alternate files to the running version.
This inserts required statements before and after LOCAL_RULESETS, you may add more statements that belong here.
Main items D{Paranoid}"%setup{paranoid}" paranoid level set above D{mail8yhabr}"YOU HAVE ALREADY BEEN REFUSED!" D{mail8ctboood}"SPAMMER CLAIMED TO BE ONE OF OUR DOMAINS!" D{mail3tt}"ONLY MAIL TO SUPPLIED Trouble Ticket ACCEPTED"
HELO DOMAIN NAME CHECKING
Most SPAMMERS use ZOMBIE PC's to send their spam, most if not all have completly numeric DNS names.
Most have their IP address in simple dotted or dashed notation, often all 4 parts of their IP address, we will not let any who have 2 or more parts of their address as their name through.
We are considering a tuning element to vary this, maybe to just one, however a lot of real senders with several servers use the last part of the IP address in their name.
Some unhelpfull ISP's string the IP's together as a single number.
Some very unhelpfull ISP's encode the numbers in HEX.
And finally totally random strings of numbers and letters, which have led us in the past to completly block the entire domain in the standard access.db file, this is often the best thing to do with hard to otherwise stop SPAMMER domains.
As much as possible of the code is INLINED to reduce the "named rulesets" total, as inlined code must be defined before use, macros are in reverse order.
[Pad,Hex,PadHex]Number convert single digits to what the name sugests.
(Pad,Hex,PadHex)IpNumber
convert four digit IP address to what the name sugests.
ScreenMash
The worker, matchs supplied patten with $s (HELLO).
However Hello must be clearly tokenised.
Splice used by ScreenIP
There must be a better way to do this, however as far as decimal numerics goes this works, nothing todate (lots of time spent trying) works for HEX.
celmorlauren will continue to use the original Perl helpers for now.
ScreenIpPatten used by above, trys patten dotted then dashed
ScreenIP
used by above, trims IP address from 4 then 3 then 2, also trys re-arranging and all 4 parts spliced together as a single token
ScreenDomainIP small often used macro, re-arranges IP to check
CONTACT
This bit arrived at on first contact, and so permissions based on IP can be set
Local_check_relay standard rule, to check incoming connection against mail8 databases and of course standard local ip addresses, further rules are based on what happens here.
uses Macro Screem_bad_relay (GLOBAL B) to do the main checking
{GoodRelay} and {BadRelay} both contain result of check, such values as (where # is checked value).
#.Local.FOUND $w {GoodRelay} #.VirtHost.FOUND ${VirtHost} {GoodRelay} #.RelayDomain.FOUND $R {GoodRelay} #.mail1.FOUND mail1.db {GoodRelay} #.Private.FOUND 192.168.#.# 172.16.#.# 10.#.#.# {GoodRelay} #.mail4.FOUND #.mail3.FOUND
mail8 amd mail9 checks result im $#error
Being found does not mean that the host is a BadRelay, just that it will need handling differently to other hosts. Hosts recorded as being GoodRelay.
yahoo and hotmail hosts are recorded as {RestrictedHosts}, to help to ensure that e-mail purporting to come from these domains does infact come from these domains.
HELO & FROM
After intial HELO and every FROM following
This insists that the HELO host name must either be the same as the {client_name} or resolve to an address that is the same as the {client_name}.
This insists that mail purporting to come from hotmail or yahoo does come from the relevant domain. The mail addres is recorded by {RestrictedHost}
This also handles empty FROM's which are normally bounces of some kind, or the un-helpfull callback verify sudo bounce, which often originates from poorly configured e-mail systems that blindly bounce back to Forged FROM addresses.
{Bounce} records that a empty FROM has been recieved, these are accepted according to the value of {Paranoid}.
{Refused} and {RefusedAgain} record that the connection has been refused, only spammers will cause {RefusedAgain} to be generated, also if the Perl Helpers are installed these will attempt to ammend both sendmail databases and the firewall rules.
Refers to
this is called regardless of wether the PerlHelpers have been installed.
this checks the HELO host for being highly numeric, and having its IP encoded in the name.
RCPT
standard sendmail rules do most of the work, however depending on the value {Paranoid} responses will vary in responce to {Bounce} requests.
DATA
This bit is for checking for callback verify requests which are a fake bounce with no DATA, action depends on setting of {Paranoid} and {Bounce}
For all values of {Paranoid} other than 0, this will not accept any bounces from any unknown systems.
{Bounce} will not be defined if mail is from permitted systems.
HEADER LINES
All this does at the moment is check the following header statements
Received:
HTTP webmail is so insecure and open to abuse, that we have taken the position that we will no longer accept mail from systems that have received mail from a system that received mail vi HTTP.
Those systems already in the mail4 database (relays who fail one or more tests, and who you want to recieve mail from) are excempt from this, and so even if they have received mail with HTTP it will be accepted.
Otherwise this is controlled by the level of {Paranoid} 0 or 1 will accept, otherwise not.
some spammers pass other tests but show themselves by pretending to send from one of our domains!
From:
ALL users must conform, they must not FAKE their "From:" header to show something other than the FROM used in the mail discussion with the mail host. No exceptions! A considerable amount of SPAM comes from mailers that allow their users to send SPAM. It is in your own intrest to stop your users sending SPAM as you are very likly to be registered as a SPAMMER and be blocked by mail servers the world over.
Reply-to:
This uses the standard Sendmail access database to check that the reply is not to to an address that you would not accept mail from.
The entry in the access file would be
From:##### ERROR:"550 We do not accept mail from you"
Where ##### is tha banned server address. Many spammers are using yahoo addresses, from systems that have nothing todo with yahoo! So From: is not really the best way to stop these.
Where e-mail has been received from a {RestrictedHost} such as yahoo or hotmail, the Reply-to: must be the same as f.
Where a Reply-to: of hotmail or yahoo is used by anyone else it is refused!
Other tests are possible. But on balance we think that these should be left to a 2nd level e-mail system that can take a closer look with both anti virus and something like SpamAssassin, it should be noted that these systems tend to be rather slow, so should never be run on a busy front line e-mail system under constant attack. If you wish additional rules may be supplied, these will be tacked on the end of the SScreenHeader definition.
Versions
Nov 2006 1st version, pure sendmail M4 hack, using plug-in Perl programs.
25 August, this 1st CPAN M4 hack script, original script split into, this Mail8 and Utils for creation and testing.
21 Sept 2007 Released onto CPAN.
Amendments to release version
Most changes are recorded in Utils as this really was a messy hack conversion, with many revisions required to get something that worked.
added this HISTORY
mail8 DataBase file refrences (apart from mail4) have been removed, and will now only added if the {Perl_Helpers} have been installed, currently these scripts are not available, their future will be decided later.
mail8_zombie now included in distro, install it to get the full benifits
22 September 2007 CPAN Release
sendmail works fine in command line test mode. But complains of "too many long names" when this is installed and run, "sendmail" assigns on the "fly" macro names!????!!!!?.
A limit of 96 (8 Bit limits?) is really too constrictive.
1st try now using {MashTemp} instead of {MashStack}, saves some, so sendmail falls over a little later!
OPTION NOMASH used where possible, also "OPTION MASH 1" to conserve name space in other places, this and changes to Utils have removed 20 names
23 September 2007 CPAN Release
on SuSE 10.1 (remastered) sendmail version 8.13.6, complained of an unbalanced >, not the fault of the sender but a macro? why did sendmail then decide to 553???, had no problem with the same code on SuSE 9.3 sendmail 8.13.3?
So a single line added and hopefully all will be well, and this system can finally protect celmorlauren, we suffer quite a lot of spam, so we should be able to spot any other weakness in the code.
it appears we have caused a problem with the changes above, HELO's are failing when they should not. added failed hosts to test data, to try and ensure it does not happen again.
24 September 2007 CPAN Release
Patch required, spammer whose domain matched who they said they were, was not put through the numeric zombie check as expected, a Fix broke the intended logic, test data added to mail8.
24 September 2007 CPAN Patch Release
HELO checking for {daemon_addr} added, should have there already (whoops sorry), however numeric addresses will not resolve anyway, so CPAN patch release delayed untill more serious problems encountered. Develelpment on test_cgi will cause further amendments, so will wait for that before release.
ref to "network associates" changed to "network solutions", just been amending my "domains", so spotted the error, as I tend to look for them using google. Other typo's will be corrected when found.
Noted rather more spam than normal has been arriving at my mail system, all fake MX's have been stopped as expected, but a few from Open Relays get through this (the 1st level filter), only to fail at the 2nd level email system. In an attempt to allow the 2nd level email system to sleep more, and only have to deal with real e-mail.
Header From: checked is the same as f or at least contains f. Failures are {Refused}, no exceptions all users are checked!
Header Recieved: check now sets {Refused} for failures. {GoodRelay} hosts are exempted from this check.
Found during testing, that mail only worked once only! corrections made to ensure essential macros are set up again, and others use sendmail macros such as f instead of supplied values, which may be the from address 1st time round, 2nd and more times it is (conneted as HOST & IP)?!
30 September 2007 CPAN Patch Release
noted that the Pause upload server falls foul of the just uploaded system, so the mail4.db will now permit those listed to skip all checks, however local users will not be allowed to skip mail FROM checks.
sendmail needs more free "long names" for its own use, this had been using only 21 "long names", Sendmail::M4::Utils has been modified to provide routines that can use just one "long name" {MashFound}, so this will be recoded as required as the {macros} used before can no longer be direcltly accessed. Total saving minus 5, leaving 16.
Also found 2 other names, that did not need persistance, these now MashTempC & D
02 October 2007 CPAN Patch Release
noticed that header line "From:" where email address not included in < > brackets, will not match when should.
03 October 2007 CPAN Patch Release
the Received: header checking does not work, and currently nothing tried makes it work, fine in test mode. But as headers are not tokenised with sendmail 8.13, can not do anthing usefull with the supplied tools.
So currently does nothing, just returns.
Fix must wait for Sendmail::M4::mail8_daemon and its worker module Sendmail::M4::Mail8_daemon.pm, which will now do more than originally designed to do.
Patch required to remove useless error messages from the logs.
noted that dots and dashs did not work in From: header checker for names, fixed. However underscores will still not work
05 October 2007 CPAN Patch Release
Noted that lots of SPAM has Reply-to: yahoo or hotmail, system will now check sendmails standard access database for From:#### where #### is system we do not accept mail from. SPAMMERS are now concentrating on finding open-relays or other poorly setup and vunerable systems, many of which seem to be Exim servers. These are of course blocked on their third SPAM or sometimes on their first if they are an open-relay. SPAM detection being done by the second level e-mail system, the CPAN version of the interface between the two systems is currently being re-writen, and we will make it available soon.
08 October 2007 CPAN Patch Release
Noted that the bulk of spam that has got through to the second level filter, has originated from WEB mail, which will now be refused with a {Paranoid} setting of 2 or more.
Received: DOMAIN name checking for phoney received from # by # now working, at least for us at the 1st level.
yahoo and hotmail domains are the Reply-to: and From: names of choice for spammers. ALL sending from domains that have nothing to do with yahoo or hotmail claiming to relay their domains, will be now stopped. Hard coded as these are so well known!
Documentation clean-up. HISTORY moved to end of document.
12 October 2007 CPAN Patch Release
Whoops, a hotmail mail was refused too early due to a typo, strung several tests together at the end with the unlucky users connection details, problem fixed.
Corrected version will run on the mail server, before it is uploaded.
14 October 2007 CPAN Patch Release
9 POD Errors
The following errors were encountered while parsing the POD:
Expected text after =item, not a bullet
You forgot a '=back' before '=head1'
Expected text after =item, not a number
You forgot a '=back' before '=head2'
=over without closing =back
To install Sendmail::M4::Utils, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Sendmail::M4::Utils
CPAN shell
perl -MCPAN -e shell install Sendmail::M4::Utils
For more information on module installation, please visit the detailed CPAN module installation guide.