Tomas Podermanski > Net-NfDump > Net::NfDump

Download:
Net-NfDump-1.04.tar.gz

Dependencies

Annotate this POD

View/Report Bugs
Module Version: 1.04   Source  

NAME ^

Net::NfDump - Perl API for manipulating with nfdump files

SYNOPSIS ^

  use Net::NfDump;

  #
  #
  # Example 1: reading nfdump file(s)
  # 
  
  $flow = new Net::NfDump(
              InputFiles => [ 'nfdump_file1', 'nfdump_file2' ], 
              Filter => 'icmp and src net 10.0.0.0/8',
              Fields => 'proto, bytes' ); 

  $flow->query();

  while (my ($proto, $bytes) = $flow->fetchrow_array() )  {
      $h{$proto} += $bytes;
  }
  $flow->finish();

  foreach ( keys %h ) {
      printf "%s %d\n", $_, $h{$_};
  }


  #
  #
  # Example 2: creating and writing records to nfdump file
  #
  
  $flow = new Net::NfDump(
              OutputFile => 'output.nfcap',
              Fields => 'srcip,dstip' );

  $flow->storerow_arrayref( [ txt2ip('147.229.3.10'), txt2ip('1.2.3.4') ] );

  $flow->finish();


  #
  #
  # Example 3: reading/writing (merging two input files) and swap
  #            source and destination address if the destination port 
  #            is 80/http (I know it doesn't make much sense).
  #

  $flow1 = new Net::NfDump( 
               InputFiles => [ 'nfdump_file1', 'nfdump_file2' ], 
               Fields => 'srcip, dstip, dstport' ); 

  $flow2 = new Net::NfDump( 
               OutputFile => 'nfdump_file_out', 
               Fields => 'srcip, dstip, dstport' ); 

  $flow1->query();
  $flow2->create();

  while (my $ref = $flow->fetchrow_arrayref() )  {

      if ( $ref->[2] == 80 ) { 
          ($ref->[0], $ref->[1]) = ($ref->[1], $ref->[0]);
      }

     $flow2->clonerow($flow1);
     $flow2->storerow_arrayref($ref);

  }

  $flow1->finish();
  $flow2->finish();

DESCRIPTION ^

Nfdump http://nfdump.sourceforge.net/ is a very popular toolset for collecting, storing and processing NetFlow/SFlow/IPFIX data. One of the key tools is a command line utility bearing the same name as the whole toolset (nfdump). Although this utility can process data very fast, it is cumbersome for some applications.

This module implements basic operations and allows to read, create and write flow records on binary files produced with nfdump tool. The module tries to keep the same naming conventions for methods as are used in DBI modules/API, so developers who got used to work with such interface should remain familiar with the new one.

The module uses the original nfdump sources to implement necessary functions. This enables to keep the compatibility with the original nfdump quiet easily and to cope with future versions of the nfdump tool with a minimal effort.

The architecture is following:

          APPLICATION 
   +------------------------+
   |                        |  Implements all methods and functions 
   | Net::NfDump API (perl) |  described in this document.
   |                        |
   +------------------------+
   |                        |  The code converts internal nfdump 
   | libnf - glue code (C)  |  structures into perl and back to C.
   |                        |
   +------------------------+
   |                        |  All original nfdump source files. There  
   |   nfdump sources (C)   |  are no changes in these files. All  
   |                        |  changes are placed into libnf code.
   +------------------------+  
         NFDUMP FILES

This version of Net::NfDump module is based on nfdump-1.6.12 available on http://sourceforge.net/projects/nfdump/. Support for NSEL code is enabled.

WARNING FOR VERSION >= 0.13 ^

The files created by Net::NfDump version >= 0.13 can be read only with nfdump 1.6.12 and newer. For reading it supports all formats starting with nfdump 1.6.

METHODS, OPTIONS AND RELATED FUNCTIONS ^

Options

Options can be handled by various methods. The basic options can be handled by the constructor and then modified by methods such as $obj->query() or $obj->create().

The values after => indicate the default value for the item.

Constructor, status information methods

Methods for reading data

Methods for writing data

Extra conversion and support functions

The module also provides extra convertion functions which allow to convert binnary format of IP address, MAC address and MPLS labels tag into text format and back.

Those functions are not exported by default, therefore it has to be either called with full module name or imported when the module is loaded. To import all support function :all a synonym may be used.

  use Net::NfDump qw ':all';

SUPPORTED ITEMS ^

Up to date list of supported items is available on Net::NfDump::Fields

  Time items
  =====================
  first - Timestamp of the first packet seen (in miliseconds)
  last - Timestamp of the last packet seen (in miliseconds)
  received - Timestamp regarding when the packet was received by collector 

  Statistical items
  =====================
  bytes - The number of bytes 
  pkts - The number of packets 
  outbytes - The number of output bytes 
  outpkts - The number of output packets 
  flows - The number of flows (aggregated) 

  Layer 4 information
  =====================
  srcport - Source port 
  dstport - Destination port 
  tcpflags - TCP flags  

  Layer 3 information
  =====================
  srcip - Source IP address 
  dstip - Destination IP address 
  nexthop - IP next hop 
  srcmask - Source mask 
  dstmask - Destination mask 
  tos - Source type of service 
  dsttos - Destination type of service 
  srcas - Source AS number 
  dstas - Destination AS number 
  nextas - BGP Next AS 
  prevas - BGP Previous AS 
  bgpnexthop - BGP next hop 
  proto - IP protocol  

  Layer 2 information
  =====================
  srcvlan - Source vlan label 
  dstvlan - Destination vlan label 
  insrcmac - In source MAC address 
  outsrcmac - Out destination MAC address 
  indstmac - In destination MAC address 
  outdstmac - Out source MAC address 

  MPLS information
  =====================
  mpls - MPLS labels 

  Layer 1 information
  =====================
  inif - SNMP input interface number 
  outif - SNMP output interface number 
  dir - Flow directions ingress/egress 
  fwd - Forwarding status 

  Exporter information
  =====================
  router - Exporting router IP 
  systype - Type of exporter 
  sysid - Internal SysID of exporter 

  NSEL fields, see: http://www.cisco.com/en/US/docs/security/asa/asa81/netflow/netflow.html
  =====================
  eventtime - NSEL The time that the flow was created
  connid - NSEL An identifier of a unique flow for the device 
  icmpcode - NSEL ICMP code value 
  icmptype - NSEL ICMP type value 
  xevent - NSEL Extended event code
  xsrcip - NSEL Mapped source IPv4 address 
  xdstip - NSEL Mapped destination IPv4 address 
  xsrcport - NSEL Mapped source port 
  xdstport - NSEL Mapped destination port 
 NSEL The input ACL that permitted or denied the flow
  iacl - Hash value or ID of the ACL name
  iace - Hash value or ID of the ACL name 
  ixace - Hash value or ID of an extended ACE configuration 
 NSEL The output ACL that permitted or denied a flow  
  eacl - Hash value or ID of the ACL name
  eace - Hash value or ID of the ACL name
  exace - Hash value or ID of an extended ACE configuration
  username - NSEL username

  NEL (NetFlow Event Logging) fields
  =====================
  ingressvrfid - NEL NAT ingress vrf id 
  eventflag -  NAT event flag (always set to 1 by nfdump)
  egressvrfid -  NAT egress VRF ID

  NEL Port Block Allocation (added 2014-04-19)
  =====================
  blockstart -  NAT pool block start
  blockend -  NAT pool block end 
  blockstep -  NAT pool block step
  blocksize -  NAT pool block size

  Extra/special fields
  =====================
  cl - nprobe latency client_nw_delay_usec 
  sl - nprobe latency server_nw_delay_usec
  al - nprobe latency appl_latency_usec

PERFORMANCE ^

It is obvious that performance of the perl interface is lower in comparison to highly optimized nfdump utility. While nfdump is able to process up to 2 milion of records per second, the Net::NfDump is not able to process more than 1 milion. However, there are several rules to keep the code optimised:

NOTE ABOUT 32BIT PLATFORMS ^

Nfdump primary uses 64 bit counters and other items to store single integer value. However, the native 64 bit support is not compiled in every perl. For those cases where only 32 integer values are supported, the Net::NfDump uses Math::Int64 module.

The build scripts detect the platform automatically and Math::Int64 module is required only on platforms where an available perl does not support 64bit integer values.

AGGREGATION, STATISTICS AND SORTING ^

The current version of Net::NfDump do not support aggregation, statistics and sorting. This features are planned for future version. However tehere are some workarounnds how to use thoose features in Net::NfDump.

EXAMPLES OF USE ^

There are several examples in the examples directory.

download_asn_db, nf_asn_geo_update - The set of scripts for updating the information about AS numbers and country codes based on BGP and geolocation database. Every flow can be extended with src/dst AS number and src/dst country code.

The first script (download_asn_db) downloads the BGP database which is available on RIPE server. Then, the database is preprocessed and prepared for the second script (with support of Net::IP::LPM module).

The second script (download_asn_db) updates the AS (or country code) information in the nfdump file. It can be run as the extra command (-x option of nfcapd) to update information when the new file is available.

The information about src/dst country works in a similar way. It uses maxmind database and Geo::IP module. However, nfdump does not support any field to store such kind of information; the xsrcport and xdstport fields are used instead. The country code is converted into 16 bit information (8 bits for the first character of a country code and another 8 bits for the second one).

SEE ALSO ^

http://nfdump.sourceforge.net/

AUTHOR ^

Tomas Podermanski, <tpoder@cis.vutbr.cz>, Brno University of Technology

COPYRIGHT AND LICENCE ^

Copyright (C) 2012 by Brno University of Technology

This library is free software; you can redistribute it and modify it under the same terms as Perl itself.

If you are satisfied with using Net::NfDump, please, send us a postcard, preferably with a picture of your location / city to:

  Brno University of Technology 
  CVIS
  Tomas Podermanski 
  Antoninska 1
  601 90 
  Czech Republic 
syntax highlighting: