NAME
ThreatNet::DATN2004 - Proposal: The Decentralised Active Threat Network
This document has been created to describe a concept that may be of use
in a variety of fields. It should be considered a general concept only
and is subject to change.
This CPAN/POD version of the document, first published in December 2004
at <http://ali.as/devel/threatnetwork.html>, has been released to
independantly timestamp and archive the concept and proposal in case of
future patent-related issues by companies and to attempt to keep the
core idea available to all.
INTRODUCTION
On the Internet there exists an increasing number of different ways in
which hosts are being misused or abused. Likewise there is also an
increasing number of ways in which these known-bad hosts are being
identified. Most of these occur in the process of a particular task,
such as checking an email message for spam status.
As these hosts are identified, their identify is transmitted across to
internet to members of threat networks. The most common of these are the
various email "black lists", most of which use DNS or some other method
to publish lists of known-bad ips or ip ranges. Mail processing services
submit requests to a DNS server storing these lists to determine if a
particular host contacting them is a known spammer.
This draft specification describes a system which would be used to
identify a specific category of these bad ips, hosts that can be
considered "Active Threats". An Active Threat is a host that is
currently engaged in anti-social, damaging or criminal behaviour, such
as actively sending out spam or viruses. This specification is NOT
intended to deal with long-term offenders, as they are addressed by a
number of current systems.
If applied to long term offenders, hosts would be registered as an
Active Threat when they commence their anti-social behaviour, and fall
off any list during periods in which they are not conducting this
behaviour.
The general intent is to deal with only those hosts that are actively
engaged in damaging behaviour, whether or not they are long term
offenders or new offenders. And to deal with the hosts as soon as
possible, ideally within a few seconds.
GENERAL PRINCIPLES
The following general principles have been establish for this system,
and guide the sample implementation described below. They are listed
(loosely) in priority order.
Speed of Response
The ThreatNet concept is specifically intended to address active and
transient threats. If a Verified Threat is detected at time $t, other
members of the same Threat Network should start to recieve notification
before time $t + 1 (within 1 second). Full propagation to all members,
and any subsequent responses, should be complete within by $t + 60
(within 1 minute).
Within a second of a host being detected conducting some anti-social
behaviour, the member should be able to confirm this behaviour and the
host ip responsible for it, and classify the host as a Verified Threat,
issuing a message into a Threat Network. Notification to all members of
the Threat Network and any other linked networks should occur as quickly
as the Internet and Threat Network itself is able to spread this
information. Any members who wish to respond to the threat, or are able
to neutralise it, should be able to act within a few seconds and
certainly within a minute.
For the example of spam, this would allow newly compromised hosts
("zombies") acting as spam or virus agents that are not on current block
lists to be dealt with and blocked before they are able to send
significant amounts of spam. This could also allow for compromised hosts
on dynamic IPs to be blocked each time they move IPs without
significantly damaging subsequent users of the IPs, or needing to block
the entire IP range. This could also help to reduce the impact of new
and fast-moving viruses, as any newly infected computers appearing
within properly managed ISPs could be disconnected and disabled before
they are able to make a significant number of attempts to reproduce.
Flexibility and Integration
The implementation should be as flexible as possible and integrate with
many other current systems, or provide a way for developers to do this
integration themselves. Although the initial idea for this concept was
as an anti-spam system, it should not be considered solely an anti-spam
tool, and should be able to be easily reconfigured for use in any other
scenario that involves similar sorts of rapid responses to threats.
The entire system should be defined as a set of separate components, any
of which can be replaced, improved or set up in a variety of different
ways. Each function or role should be identified separately so that new
systems and implementations can provide one particular role without
needing to implement all of them.
Distributed and Decentralised
Most existing anti-spam or anti-threat networks involve a central
authorative data source, and a central authorative distribution network
to spread it, typically a managed database of some sort and a DNS/rsync
combination. Centralising in this way can lead to the blocking systems
themselves becoming the target of both technical and legal attacks.
Extreme measures are often required to protect (technically and legally)
the blocking systems.
This proposal involves a decentralised system with no long term state
information and no central server or service that can be implemented in
an ad-hoc way. Having limited the scope of the network to short term and
transient threat this can be done relatively easily.
Short term state information is held locally at each node, and a mandory
"quiet period" is enforced when joining a new network to let the new
member synchronise with the network.
This quiet period will keep load and communication volume down, and let
new members join a network without the need for a synchronisation event.
By waiting for the network's block period before contributing, each node
can be sure that it has fully synchronised with the network before
beginning to contribute block entries that might otherwise be
duplicates.
Ease of Implementation
In order to get this concept off the ground and running quickly, and to
help implement it as flexibly as possible, it is proposed that the
system makes use of common, time-tested, well-supported and mature
systems wherever possible.
Minimisation of Harm
Long term blocking, and the blocking of entire subnets, may not be
suitable for dealing with new and short term threats. One bad apple can
ruin it for the entire barrel. While the ability to "poison" a Threat
Network may still exist as it does in existing blocking systems, this
poisoning would need to be active and continuous, "re-poisoning" the
Threat Network with new messages each time the short block period nears
its end.
For dynamic IP connections, any response taken against a single IP will
expire automatically within the block period, once the host stops use of
the IP.
Elegant Degradation and Failure
As with any security system, the greater and higher profile its use, the
bigger the target it becomes and more likely to be attacked. The system
should be designed from the start to be able to stand up to attack when
deployed in high-profile scenarios, and to degrade gracefully where
possible to retain at least partial functionality.
EXAMPLE IMPLEMENTATION
The following describes the set of elements which would be required to
create a single complete system, and a set of optional elements which
could be used to enhance or expand the system.
Required Element: Messaging Platform
At the core of the Threat Network is a messaging platform. As an
extremely mature and often-attacked network messaging system, IRC would
appear an ideal platform to use for this purpose. Libraries to connect
to and work with IRC already exists in almost every programming
language. In addition, it can be deployed extremely flexibly from either
a non-authenticated channel on an existing IRC network to secret and
private dedicated servers with encryption, certificate-backed
authentication, and agent-based monitoring.
Required Element: Language Specification
Each member of the network should interact in a standard and relatively
flexible language. While this specification does not attempt to describe
the details of any particular language, for example purposes only we
will start with the simplest possible language. The sample language
contains only a single message with only the IP address of a validated
threat, assumed to have been detected within a few seconds of the threat
event occurring.
Required Element: Network Agents
Within the IRC Threat Network, there exists a number of members, each
consisting of at least a software agent. Any agent connected to the
system will consist of at least the Event Listener element described
below, and one or more optional elements. At least one agent is required
to implement the Threat Provider element in order to create a working
system.
Required Element: Event Listener
A pure Event Listener is a passive agent that listens on a Threat
Network for threat messages and optionally acts upon them. Every agent
connected to the Threat Network is required to implement an Event
Listener to monitor incoming messages, if only to prevent issuing
duplicate messages.
Many of the agents that would implement only the Event Listener would
act as interfaces and adapters to other systems.
As one example, a ThreatNet to DNSBL adapter might listen for threat
messages, submit a DNSBL request against each IP, and for each IP that
isn't already listed, add it to the local DNSBL server with a short TTL
most likely matching the ThreatNet block period.
A ThreatNet to firewall adapter could listen for specific types of
high-impact threat messages and inject blocks into a firewall
configuration, allowing particular types of threats to be blocked at a
connection level.
Depending on the scale of the network, and the ability of the adapter to
handle the volume of responses, the Event Listener may need to institute
some form of queuing or flood control. It would be expected that such
flood control would become common were the popularity of Threat Networks
to grow significantly.
Optional Element: Threat Cache / Threat Filter
While some Threat Network agents do not need to maintain state, a great
many agent types do, in particular anything that wishes to submit
Verified Threat messages of its own to the network.
A Threat Cache is simply a small local database, probably in-memory,
that stores all "current" threats. A Threat Filter handles and passes on
only the subset of events that meet a certain set of criteria. For
agents joining a network, the purpose of waiting for the full block
period until beginning to submit messages is to allow the agent's Threat
Cache to fill and thus synchronise with the rest of the members of the
network.
In another example, an ISP may deploy a "Zombie Response" agent that
applies a filter to limit the event feed to only those Verified Threats
that are the inside the ISP's own network. When the agent identifies an
infected or trojaned customer by IP, it could terminate the connection,
disable the login, and flag the customer's account so that when they try
to reconnect, they will be informed they are infected with a virus or
that someone may be using their computer without their permission.
Optional Element: Threat Provider
A Threat Provider is an agent that submits Verified Threats messages to
a network. Because any message that is added to the network may result
in a variety of actions against the host, care should be taken to verify
that the host is indeed an Active Threat.
Any Threat Provider should also act as an Event Listener and ensure that
Verified Threat events are not submitted if they are already "current"
(in the Threat Cache). The default TTL on all threats will be 1 hour.
As an example, a spam "secondary MX trap" might be tied to a local
Threat Network agent that would aggregate and filter threats from the
host, and submit Veried Threat messages into the network.
By separating the Threat Network from the individual applications, new
and more accurate anti-spam scripts can incorporate a connection to the
agent and feed more advanced Verified Threat messages into the same
network, be it local, regional, or larger.
Optional Element: Network Bridge
To separate a private company-wide network, or a university network from
a larger network, you can make use of a Network Bridge. This is an agent
that connects to two or more different networks, maintains a combined
Threat Cache and feeds Verified Threat messages from one to the other,
potentially in both directions.
For example, a single or small group of universities could run a private
Threat Network for open use by members of the University, with a single
Network Bridge agent pulling in messages from a larger national or
international Threat Network. The bridge would be configured to filter
out any messages that reference hosts already listed on the university
DNSBL server.
Optional Element: Security Model
One advantage of using IRC is that any of the existing security features
that are currently part of IRC can also be applied to a threat network,
including voice right, operators, channel management bots, SIRC and
more.
At the most open, a single ThreatNet channel could be run within general
IRC server with anyone free to participate and submit any entries they
wish. This would make it trivially easy to set up an environment for
testing use.
More secure than this, a bot could run the channel allowing anyone to
join, but only giving voice permissions to agents joining from
white-listed IP addresses.
On a larger scale, a large dedicated SIRC network could be created, with
server or network-level accounts and monitoring bots to kick and issue
Verified Threat events for any anti-social accounts connected to the
Threat Network.
CONCLUSION
While not useful for long term blocking, the Decentralised Active Threat
Network would dramatically improve the response time to threats, with
the goal creating a powerful and effective "nervous system" for
detecting and dealing with spammers, virus-infected or trojaned hosts,
or other threats for which the damage could be greatly reduced by
dealing with them immediately.
GLOSSARY OF TERMS
Active Threat
Any internet host that is currently engaged in anti-social or illegal
behaviour such as spamming, virus transmission or acting as part of a
DDOS or other attack.
Verified Threat
A host/ip that can be positively and reliably confirmed to be an Active
Threat.
Threat Network
An communications network, populated by software agents, for the purpose
of rapidly distributing information about, and responding to, Active
Threats.
Threat Language
A protocol or message format used to describe a Verified Threat on a
Threat Network.
Event Listener
A software agent which is connected to one or more Threat Networks and
receives Verified Threat messages from the network.
Threat Cache
A stored list of "current" Active Threats, primarily used to prevent the
flooding of duplicate messages onto the Threat Network.
Threat Filter
Any collection of IPs or IP networks against which a stream of events is
checked, removing or keeping only a certain subset of all events.
Threat Provider
An software agent which connects to a Threat Network and submits
Verified Threat messages to the network.
Network Bridge
A software agent that sits "between" two or more Threat Networks,
transmitting new Verified Threat messages between them, after checking
against an arbitrary number of Threat Filters.
Threat Response
Some action that undertaken to defend against or counter an Active
Threat, in response to a Verified Threat message. This could include
short-term DNSBL listing, firewall block entries, or actively disabling
the internet connection of the host.
PARTICIPATION
The website http://ali.as/ will provide a link to the current forums of
idea exchange for the ThreatNet project.
In addition, there is also a #threatnet channel on FreeNode for testing
implementations of the reference implementation described above.
AUTHOR
Adam Kennedy <adamk@cpan.org>
SEE ALSO
ThreatNet, ThreatNet::Message, ThreatNet::Topic
COPYRIGHT
Copyright 2004 - 2008 Adam Kennedy.
All ThreatNet modules on CPAN are generally free software; you can
redistribute them and/or modify them under the same terms as Perl
itself.
The full text of the license can be found in the LICENSE file included
with this documentation.