Net::NfDump - Perl API for manipulating with nfdump files
use Net::NfDump; # # # Example 1: reading nfdump file(s) # $flow = new Net::NfDump( InputFiles => [ 'nfdump_file1', 'nfdump_file2' ], Filter => 'icmp and src net 10.0.0.0/8', Fields => 'proto, bytes' ); $flow->query(); while (my ($proto, $bytes) = $flow->fetchrow_array() ) { $h{$proto} += $bytes; } $flow->finish(); foreach ( keys %h ) { printf "%s %d\n", $_, $h{$_}; } # # # Example 2: creating and writing records to nfdump file # $flow = new Net::NfDump( OutputFile => 'output.nfcap', Fields => 'srcip,dstip' ); $flow->storerow_arrayref( [ txt2ip('147.229.3.10'), txt2ip('1.2.3.4') ] ); $flow->finish(); # # # Example 3: reading/writing (merging two input files) and swap # source and destination address if the destination port # is 80/http (I know it doesn't make much sense). # $flow1 = new Net::NfDump( InputFiles => [ 'nfdump_file1', 'nfdump_file2' ], Fields => 'srcip, dstip, dstport' ); $flow2 = new Net::NfDump( OutputFile => 'nfdump_file_out', Fields => 'srcip, dstip, dstport' ); $flow1->query(); $flow2->create(); while (my $ref = $flow->fetchrow_arrayref() ) { if ( $ref->[2] == 80 ) { ($ref->[0], $ref->[1]) = ($ref->[1], $ref->[0]); } $flow2->clonerow($flow1); $flow2->storerow_arrayref($ref); } $flow1->finish(); $flow2->finish();
Nfdump http://nfdump.sourceforge.net/ is a very popular toolset for collecting, storing and processing NetFlow/SFlow/IPFIX data. One of the key tools is a command line utility bearing the same name as the whole toolset (nfdump). Although this utility can process data very fast, it is cumbersome for some applications.
This module implements basic operations and allows to read, create and write flow records on binary files produced with nfdump tool. The module tries to keep the same naming conventions for methods as are used in DBI modules/API, so developers who got used to work with such interface should remain familiar with the new one.
The module uses the original nfdump sources to implement necessary functions. This enables to keep the compatibility with the original nfdump quiet easily and to cope with future versions of the nfdump tool with a minimal effort.
The architecture is following:
APPLICATION +------------------------+ | | Implements all methods and functions | Net::NfDump API (perl) | described in this document. | | +------------------------+ | | The code converts internal nfdump | libnf - glue code (C) | structures into perl and back to C. | | +------------------------+ | | All original nfdump source files. There | nfdump sources (C) | are no changes in these files. All | | changes are placed into libnf code. +------------------------+ NFDUMP FILES
This version of Net::NfDump module is based on nfdump-1.6.12 available on http://sourceforge.net/projects/nfdump/. Support for NSEL code is enabled.
The files created by Net::NfDump version >= 0.13 can be read only with nfdump 1.6.12 and newer. For reading it supports all formats starting with nfdump 1.6.
Options can be handled by various methods. The basic options can be handled by the constructor and then modified by methods such as $obj->query() or $obj->create().
The values after => indicate the default value for the item.
InputFiles => []
List of files to read (arrayref).
Filter => 'any'
Filter that is applied on input records. It uses nfdump/tcpdump syntax.
Fields => '*'
List of fields to read or to update. Any supported field can be used here. See the chapter "Supported Fields" for the full list. Special field * can be used to define all fields.
Aggreg => 0
Create aggregated result. When the method ->query() is callet the library loads data into memory structure and perform aggregation according the Fields attribute.
OrderBy => '<none>'
Sort the final result according the field specified. It can by used only for aggregated results.
TimeWindowStart, TimeWindowEnd => 0
Filter flows that start or end in the specific time window. The options use unix timestamp values or 0 if the filter should not be applied.
OutputFile => undef
Output file for storerow_* methods. Default: undef
Compressed => 1
Flag indicating whether the output files should be compressed or not.
Anonymized => 0
Flag indicating that output file contains anonymized data.
Ident => ''
String identificator of files. The value is stored in the file header.
$obj = new Net::NfDump( %opts )
my $obj = new Net::NfDump( InputFiles => [ 'file1'] );
The constructor. It defines the way the parameter options can be specified.
$ref = $obj->info()
my $i = $obj->info(); print Dumper($i);
informs about the current state of processing input files. It returns information about already processed files, blocks and records. The information may be useful for estimating the time of processing the whole dataset. Hashref returns following items:
total_files - total number of files to process elapsed_time - elapsed time remaining_time - estimated remaining time to process all records percent - estimated percentage of processed records processed_files - total number of processed files processed_records - total number of processed records processed_blocks - total number of processed blocks processed_bytes - total number of processed bytes number of bytes read from file system after uncompressing current_filename - the name of the file currently processed current_total_blocks - the number of blocks in the currently processed file current_processed_blocks - the number of processed blocks in the currently processed file
$obj->finish()
$obj->finish();
closes all open file handles. It is necessary to call the method especially when a new file is created. The method flushes the file records which remained in the memory buffer and updates file statistics in the header. Without calling this method the output file might be corrupted.
$obj->query( %opts )
$obj->query( Filter => 'src host 10.10.10.1' );
This method has to be applied before any of the fetchrow_* methods is used. Any option described before can be used as a parameter of the method.
fetchrow_*
$ref = $obj->fetchrow_arrayref()
while (my $ref = $obj->fetchrow_arrayref() ) { print Dumper($ref); }
This method has to be used after the query method. The method $obj->query() is called automatically if it has not been called before.
It returns array reference with the record and skips to next record. It returns "true" if there are more records to read or "undef" if the end of a record set has been reached.
@array = $obj->fetchrow_array()
while ( @array = $obj->fetchrow_arrayref() ) { print Dumper(\@array); }
It has the same function as fetchrow_arrayref; however, it returns items in array instead.
$ref = $obj->fetchrow_hashref()
while ( $ref = $obj->fetchrow_hashref() ) { print Dumper($ref); }
The same case as fetchrow_arrayref; however, the items are returned in the hash reference as the key => vallue tuples.
NOTE: This method can be very ineffective in some cases, please, see PERFORMANCE section.
$obj->create( %opts )
$obj->create( OutputFile => 'output.nfcapd' );
This method creates a new nfdump file and has to be applied before any of $obj->storerow_* method is called.
$obj->storerow_arrayref( [ @array ] )
$obj->storerow_arrayref( [ $srcip, $dstip ] );
The method inserts data defined in arrayref to the file opened by the method $obj->create(). The number of fields and their order have to follow the order defined in the Fields option handled during $obj->new() or $obj->create() method.
$obj->storerow_array( @array )
$obj->storerow_array( $srcip, $dstip );
The same case as storerow_arrayref; however, the items are handled as a single array.
$obj->storerow_hashref ( \%hash )
$obj->storerow_hashref( { 'srcip' => $srcip, 'dstip' => $dstip } );
It inserts the structure defined as hash reference into output file.
$obj->clonerow( $obj2 )
$obj->clonerow( $obj2 );
This method copies the full content of the row from the source object (instance). This method is useful for writing effective scripts. See above the PERFORMANCE chapter.
The module also provides extra convertion functions which allow to convert binnary format of IP address, MAC address and MPLS labels tag into text format and back.
Those functions are not exported by default, therefore it has to be either called with full module name or imported when the module is loaded. To import all support function :all a synonym may be used.
:all
use Net::NfDump qw ':all';
$txt = ip2txt( $bin )
$bin = txt2ip( $txt )
$ip = txt2ip('10.10.10.1'); print ip2txt($ip);
Converts both IPv4 and IPv6 addresses into text form and back. The standard inet_ntop/inet_pton functions can be used instead to provide the same results.
Function txt2ip returns binnary format of IP address or "undef" if the conversion is not possible.
$txt = mac2txt( $bin )
$bin = txt2mac( $txt )
$mac = txt2mac('aa:02:c2:2d:e0:12'); print mac2txt($mac);
It converts MAC address to xx:yy:xx:yy:xx:yy format and back. The function mac2txt accepts an address of any following format:
aabbccddeeff aa:bb:cc:dd:ee:ff aa-bb-cc-dd-ee-ff aabb-ccdd-eeff
It returns the binnary format of an address or "undef" if the conversion is not possible.
$txt = mpls2txt( $mpls )
$mpls = txt2mpls( $txt )
$mpls = txt2mpls('1002-6-0 1003-6-0 1004-0-1'); print mpls2txt($mpls);
It converts label information into format Lbl-Exp-S and back.
Where:
Lbl - Value given to the MPLS label by the router. Exp - Value of the experimental bit. S - Value of the end-of-stack bit: Set to 1 for the oldest entry in the stack and to zero for all other entries.
$ref = flow2txt( \%row )
$ref = txt2flow( \%row )
The function flow2txt gets hash reference to the items returned by fetchrow_hashref and converts all items into text format readable for human. It applies functions ip2txt, mac2txt, mpl2txt to the items for which it makes sense. The function txt2flow does the exact opossite.
$ref = file_info( $file_name )
$ref = file_info('file.nfcap'); print Dumper($ref);
It reads information from the nfdump file header and provides various attributes such as number of blocks, version, flags, statistics, etc. As the result, the following items are returned:
version ident blocks catalog anonymized compressed sequence_failures first last flows, bytes, packets flows_tcp, flows_udp, flows_icmp, flows_other bytes_tcp, bytes_udp, bytes_icmp, bytes_other packets_tcp, packets_udp, packets_icmp, packets_other
Up to date list of supported items is available on Net::NfDump::Fields
Time items ===================== first - Timestamp of the first packet seen (in miliseconds) last - Timestamp of the last packet seen (in miliseconds) received - Timestamp regarding when the packet was received by collector Statistical items ===================== bytes - The number of bytes pkts - The number of packets outbytes - The number of output bytes outpkts - The number of output packets flows - The number of flows (aggregated) Layer 4 information ===================== srcport - Source port dstport - Destination port tcpflags - TCP flags Layer 3 information ===================== srcip - Source IP address dstip - Destination IP address nexthop - IP next hop srcmask - Source mask dstmask - Destination mask tos - Source type of service dsttos - Destination type of service srcas - Source AS number dstas - Destination AS number nextas - BGP Next AS prevas - BGP Previous AS bgpnexthop - BGP next hop proto - IP protocol Layer 2 information ===================== srcvlan - Source vlan label dstvlan - Destination vlan label insrcmac - In source MAC address outsrcmac - Out destination MAC address indstmac - In destination MAC address outdstmac - Out source MAC address MPLS information ===================== mpls - MPLS labels Layer 1 information ===================== inif - SNMP input interface number outif - SNMP output interface number dir - Flow directions ingress/egress fwd - Forwarding status Exporter information ===================== router - Exporting router IP systype - Type of exporter sysid - Internal SysID of exporter NSEL fields, see: http://www.cisco.com/en/US/docs/security/asa/asa81/netflow/netflow.html ===================== eventtime - NSEL The time that the flow was created connid - NSEL An identifier of a unique flow for the device icmpcode - NSEL ICMP code value icmptype - NSEL ICMP type value xevent - NSEL Extended event code xsrcip - NSEL Mapped source IPv4 address xdstip - NSEL Mapped destination IPv4 address xsrcport - NSEL Mapped source port xdstport - NSEL Mapped destination port NSEL The input ACL that permitted or denied the flow iacl - Hash value or ID of the ACL name iace - Hash value or ID of the ACL name ixace - Hash value or ID of an extended ACE configuration NSEL The output ACL that permitted or denied a flow eacl - Hash value or ID of the ACL name eace - Hash value or ID of the ACL name exace - Hash value or ID of an extended ACE configuration username - NSEL username NEL (NetFlow Event Logging) fields ===================== ingressvrfid - NEL NAT ingress vrf id eventflag - NAT event flag (always set to 1 by nfdump) egressvrfid - NAT egress VRF ID NEL Port Block Allocation (added 2014-04-19) ===================== blockstart - NAT pool block start blockend - NAT pool block end blockstep - NAT pool block step blocksize - NAT pool block size Extra/special fields ===================== cl - nprobe latency client_nw_delay_usec sl - nprobe latency server_nw_delay_usec al - nprobe latency appl_latency_usec
It is obvious that performance of the perl interface is lower in comparison to highly optimized nfdump utility. While nfdump is able to process up to 2 milion of records per second, the Net::NfDump is not able to process more than 1 milion. However, there are several rules to keep the code optimised:
Use $obj->fetchrow_arrayref() and $obj->storerow_arrayref() instead of *_array and *_hashref equivalents. Arrayref handles only the reference to the structure with data. Avoid using *_hashref functions, it can be 5-times slower.
$obj->fetchrow_arrayref()
$obj->storerow_arrayref()
*_array
*_hashref
Handle to the perl API only items which are necessary to be used in the code. It is always more effective to define in Fields => 'srcip,dstip,...' instead of in Fields => '*'.
Fields => 'srcip,dstip,...'
Preference to using $obj->clonerow($obj2) method is highly recommended. This method copies data between two instances directly in the C code in the libnf layer.
$obj->clonerow($obj2)
Following code:
$obj1->exec( Fields => '*' ); $obj2->create( Fields => '*' ); while ( my $ref = $obj1->fetchrow_arrayref() ) { # do something with srcip $obj2->storerow_arrayref($ref); }
can be written in a more effective way (several times faster):
$obj1->exec( Fields => 'srcip' ); $obj2->create( Fields => 'srcip' ); while ( my $ref = $obj1->fetchrow_arrayref() ) { # do something with srcip $obj2->clonerow($obj1); $obj2->storerow_arrayref($ref); }
Nfdump primary uses 64 bit counters and other items to store single integer value. However, the native 64 bit support is not compiled in every perl. For those cases where only 32 integer values are supported, the Net::NfDump uses Math::Int64 module.
Net::NfDump
Math::Int64
The build scripts detect the platform automatically and Math::Int64 module is required only on platforms where an available perl does not support 64bit integer values.
The current version of Net::NfDump do not support aggregation, statistics and sorting. This features are planned for future version. However tehere are some workarounnds how to use thoose features in Net::NfDump.
Implement functions as part of perl code. However this approach might lead to very low performance.
Preprocess the data with nfdump utility and write output into separate file (in nfdump format, -w option). For exmaple:
nfdump -R 12 -w out -A dstport "dst net 147.229.0.0/16"
Then read data in Net::NfDump from out.
There are several examples in the examples directory.
examples
download_asn_db, nf_asn_geo_update - The set of scripts for updating the information about AS numbers and country codes based on BGP and geolocation database. Every flow can be extended with src/dst AS number and src/dst country code.
download_asn_db
nf_asn_geo_update
The first script (download_asn_db) downloads the BGP database which is available on RIPE server. Then, the database is preprocessed and prepared for the second script (with support of Net::IP::LPM module).
Net::IP::LPM
The second script (download_asn_db) updates the AS (or country code) information in the nfdump file. It can be run as the extra command (-x option of nfcapd) to update information when the new file is available.
The information about src/dst country works in a similar way. It uses maxmind database and Geo::IP module. However, nfdump does not support any field to store such kind of information; the xsrcport and xdstport fields are used instead. The country code is converted into 16 bit information (8 bits for the first character of a country code and another 8 bits for the second one).
Geo::IP
http://nfdump.sourceforge.net/
Tomas Podermanski, <tpoder@cis.vutbr.cz>, Brno University of Technology
Copyright (C) 2012 by Brno University of Technology
This library is free software; you can redistribute it and modify it under the same terms as Perl itself.
If you are satisfied with using Net::NfDump, please, send us a postcard, preferably with a picture of your location / city to:
Brno University of Technology CVIS Tomas Podermanski Antoninska 1 601 90 Czech Republic
1 POD Error
The following errors were encountered while parsing the POD:
=back without =over
To install Net::NfDump, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Net::NfDump
CPAN shell
perl -MCPAN -e shell install Net::NfDump
For more information on module installation, please visit the detailed CPAN module installation guide.