Linux Certif

Toute la documentation sur la certification Linux LPI

nfdump

NAME

nfdump - netflow display and analyze program

SYNOPSIS

nfdump [options] [filter]

DESCRIPTION

nfdump

is the netflow display and analyzing program of the nfdump tool set. It reads the netflow data from files stored by nfcapd and processes the flows according the options given. The filter syntax is comparable to tcpdump and extended for netflow data. Nfdump can also display many different top N flow and flow element statistics.

OPTIONS

-r inputfile

Read input data from inputfile. Default is read from stdin.

-R expr

Read input from a sequence of files in the same directory. expr may be one of:

/any/dir Read recursively all files in directory dir.
/dir/file Read all files beginning with file.
/dir/file1:file2 Read all files from file1 to file2.

When using in combination with a sub hierarchy:
/dir/sub1/sub2/file1:sub3/sub4/file2
Read all files from sub1/sub2/file1 sub3/sub4/file2 iterating over all required hierarchy levels.

Note: files are read in alphabetical sequence.

-M expr

Read input from multiple directories. expr looks like: /any/path/to/dir1:dir2:dir3 etc. and will be expanded to the directories: /any/path/to/dir1, /any/path/to/dir2 and /any/path/to/dir3 Any number of colon separated directories may be given. The files to read are specified by -r or -R and are expected to exist in all the given directories. The options -r and -R must not contain any directory part when used in conjunction with -M.

-m

Sort the netflow records according the date first seen. This option is usually only useful in conjunction with -M, when netflow records are read from different sources, which are not necessarily sorted.

-w outputfile

If specified writes binary netflow records to outputfile ready to be processed again with nfdump. The default output is ASCII on stdout.

-f filterfile

Reads the filter syntax from filterfile. Note: Any filter specified directly on the command line takes precedence over -f.

-t timewin

Process only flows, which fall in the time window timewin, where timewin is YYYY/MM/dd.hh:mm:ss[-YYYY/MM/dd.hh:mm:ss]. Any parts of the time spec may be omitted e.g YYYY/MM/dd expands to YYYY/MM/dd.00:00:00-YYYY/MM/dd.23:59:59 and processes all flow from a given day. The time window may also be specified as +/- n. In this case it is relativ to the beginning or end of all flows. +10 means the first 10 seconds of all flows, -10 means the last 10 seconds of all flows.

-c num

Limit number of records to process to the first num flows.

-a

Aggregate netflow data. Aggregation is done at connection level.

-A fields[/netmask]

Aggregate netflow data using the specified fields, where fields is a ',' separated list out of proto srcip dstip srcport dstport srcas dstas. The default is using all fields: proto,srcip,dstip,srcport,dstport. An additional netmask may be given. In that case flows from the same subnets are aggregated. In order to do proper aggregation, the IP version is important, for which the mask applies. Therefore the IP protocol version must be given in the form of: srcip4/24 for IPv4 or srcip6/64 for IPv6 address aggregation. Apply the protocol version for dstip respectively.

-I

Print flow statistics from file specified by -r, or timeslot specified by -R/-M. The printed information corresponds to pre nfdump 1.5 nfcapd stat files.

-D dns

Set dns as nameserver to lookup hostnames.

-S

Compatibility option with pre 1.4 nfdump. Is equal to -s record/packets/bytes.

-s statistic[:p][/orderby]

Generate the Top N flow or flow element statistic. statistic can be:

record Statistic about arregated netflow records.
srcip Statistic about source IP addresses
dstip Statistic about destination IP addresses
ip Statistic about any (source or destination) IP addresses
srcport Statistic about source ports
dstport Statistic about destination ports
port Statistic about any (source or destination) ports
tos Statistic about type of service
srcas Statistic about source AS numbers
dstas Statistic about destination AS numbers
as Statistic about any (source or destination) AS numbers
inif Statistic about input interface
outif Statistic about output interface
proto Statistic about IP protocols

By adding :p to the statistic name, the resulting statistic is splitted up into transport layer protocols. Default is transport protocol independant statistics.
orderby is optional and specifies the order by which the statistics is ordered and can be flows, packets, bytes, pps, bps or bpp. You may specify more than one orderby which results in the same statistic but ordered differently. If no orderby is given, statistics are ordered by flows. You can specify as many -s flow element statistics on the command line for the same run.
Example: -s srcip -s ip/flows -s dstport/pps/packets/bytes -s record/bytes

-O orderby

Specifies the default orderby for flow element statistics -s, which applies when no orderby is given at -s. orderby can be flows, packets, bytes, pps, bps or bpp. Defaults to flows.

-l [+/-]packet_num

Limit statistics output to those records above or below the packet_num limit. packet_num accepts positive or negative numbers followed by 'K' , 'M' or 'G' 10E3, 10E6 or 10E9 flows respectively. See also note at -L

-L [+/-]byte_num

Limit statistics output to those records above or below the byte_num limit. byte_num accepts positive or negative numbers followed by 'K' , 'M' or 'G' 10E3, 10E6 or 10E9 bytes respectively. Note: These limits only apply to the statistics and aggregated outputs generated with -a -s or -S. To filter netflow records by packets and bytes, use the filter syntax 'packets' and 'bytes' described below.

-n num

Define the number for the Top N statistics. Defaults to 10. If 0 is specified the number is unlimited.

-o format

Selects the output format to print flows or flow record statistics (-s record). The following formats are available:

raw Print each file flow record on multiple lines.
line Print each flow on one line. Default format.
long Print each flow on one line with more details
extended Print each flow on one line with even more details.
pipe Machine readable format: Print all fields '|' separated.
fmt:format User defined output format.

For each defined output format except -o fmt:<format> an IPv6 long output format exists. line6, long6 and extended6. See output formts below for more information.

-K key

Anonymize all IP addresses using the CryptoPAn (Cryptography-based Prefix-preserving Anonymization) module. The key is used to initialize the Rijndael cipher. key is either a 32 character string, or a 64 hex digit string starting with 0x. Anonymizing takes place after applying the flow filter, but before printing the flow or writing the flow to a file.

See http://www.cc.gatech.edu/computing/Telecomm/cryptopan/ for more information about CryptoPAn.

-q

Suppress the header line and the statistics at the bottom.

-N

Print the numbers in the summary line as plain numbers. Better parsing.

-v file

Verify file. Print data file version, number of blocks and compression status.

-z

Compress flows. Use fast LZO1X-1 compression in output file.

-Z

Check filter syntax and exit. Sets the return value accordingly.

-X

Compiles the filer syntax and dumps the filter engine table to stdout. This is for debugging purpose only.

-V

Print nfdump version and exit.

-h

Print help text on stdout with all options and exit.

RETURN VALUE

Returns

0 No error.
255 Initialization failed.
254 Error in filter syntax.
250 Internal error.

OUTPUT FORMATS

The output format raw prints each flow record on multiple lines, including all information available in the record. This is the most detailed view on a flow.

Other output formats print each flow on a single line. Predefined output formats are line, long and extended The output format line is the default output format when no format is specified. It limits the imformation to the connection details as well as number of packets, bytes and flows.

The output format long is identical to the format line, and includes additional information such as TCP flags and Type of Service.

The output format extended is identical to the format long, and includes additional computed information such as pps, bps and bpp.

Fields:

Date flow start: Start time flow first seen. ISO 8601 format including miliseconds.
Duration: Duration of the flow in seconds and miliseconds. If flows are aggregated, duration is the time span over the entire periode of time from first seen to last seen.
Proto: Protocol used in the connection.
Src IP Addr:Port: Source IP address and source port.
Dst IP Addr:Port: Destination IP address and destination port. In case of ICMP, port is decodes as type.code.
Flags: TCP flags ORed of the connection.
Tos: Type of service.
Packets: The number of packets in this flow. If flows are aggregated, the packets are summed up.
Bytes: The number of bytes in this flow. If flows are aggregated, the bytes are summed up.
pps: The calculated packets per second: number of packets / duration. If flows are aggregated this results in the average pps during this periode of time.
bps: The calculated bits per second: 8 * number of bytes / duration. If flows are aggregated this results in the average bps during this periode of time.
Bpp: The calculated bytes per packet: number of bytes / number of packets. If flows are aggregated this results in the average bpp during this periode of time.
Flows: Number of flows. If flows are listed only, this number is alwasy 1. If flows are aggregated, this shows the number of aggregated flows to one record.

Numbers larger than 1048576 (1024*1024), are scaled to 4 digits and one decimal digit including the scaling factor M, G or T for cleaner output, e.g. 923.4 M

To make the output more readable, IPv6 addresses are shrinked down to 16 characters. The seven most and seven least digits connected with two dots '..' are displayed in any normal output formats. To display the full IPv6 address, use the appropriate long format, which is the format name followed by a 6.

Example: -o line displays an IPv6 address as 2001:23..80:d01e where as the format -o line6 displays the IPv6 address in full length 2001:234:aabb::211:24ff:fe80:d01e. The combination of -o line -6 is equivalent to -o line6.

The pipe output format is intended to be read by another programm for further processing. Values are separated by a '|'. IP addresses are printed as 4 consecutive 32bit numbers. Output sequence:

Address family PF_INET or PF_INET6
Time first seen UNIX time seconds
msec first seen Mili seconds first seen
Time last seen UNIX time seconds
msec last seen Mili seconds first seen
Protocol Protocol
Src address Src address as 4 consecutive 32bit numbers.
Src port Src port
Dst address Dst address as 4 consecutive 32bit numbers.
Dst port Dst port
Src AS Src AS number
Dst AS Dst AS number
Input IF Input Interface
Output IF Output Interface
TCP Flags TCP Flags

                      000001 FIN.

                      000010 SYN

                      000100 RESET

                      001000 PUSH

                      010000 ACK

                      100000 URGENT

                      e.g. 6 => SYN + RESET
Tos Type of Service
Packets Packets
Bytes Bytes

For IPv4 addresses only the last 32bit integer is used. All others are set to zero.

The output format fmt:<format> allows you to define your own output format. A format description format consists of a single line containing arbitrary strings and format specifier as described below

%ts Start Time - first seen
%te End Time - last seen
%td Duration
%pr Protocol
%sa Source Address
%da Destination Address
%sap Source Address:Port
%dap Destination Address:Port
%sp Source Port
%dp Destination Port
%sas Source AS
%das Destination AS
%in Input Interface num
%out Output Interface num
%pkt Packets
%byt Bytes
%fl Flows
%pkt Packets
%flg TCP Flags
%tos Tos
%bps bps - bits per second
%pps pps - packets per second
%bpp bps - Bytes per package

For example the standard output format long can be created as

-o "fmt:%ts %td %pr %sap -> %dap %flg %tos %pkt %byt %fl"

You may also define your own output format and have it compiled into nfdump. See nfdump.c around line 100 for more details.

FILTER

The filter syntax is similar to the well known pcap library used by tcpdump. The filter can be either specified on the command line after all options or in a separate file. It can span several lines. Anything after a '#' is treated as a comment and ignored to the end of the line. There is virtually no limit in the length of the filter expression. All keywords are case independent.

Any filter consists of one or more expressions expr. Any number of expr can be linked together:

expr and expr, expr or expr, not expr and ( expr ).

Expr can be one of the following filter primitives:

protocol version

inet for IPv4 and inet6 for IPv6

protocol

proto <protocol> where protocol can be any known protocol such as TCP, UDP, ICMP, ICMP6 GRE, ESP, AH, or a valid protocol number.

IP address

[SourceDestination] IP <ipaddr> or
[SourceDestination] HOST <ipaddr> with <ipaddr> as any valid IPv4, IPv6 address, or a full qualified hostname. In case of a hostname, the IP address is looked up in DNS. If more than a single IP address is found, all IP addresses are chained together. ip1 or ip2 or ip3 ... The direction tag SourceDestination may be omitted.
[SourceDestination] IP IN [<iplist>]
[SourceDestination] HOST IN [<iplist>]
iplist space separated list of individual <ipaddr> or full qualified hostnames. In case of a hostname, the IP address is looked up in DNS. If more than a single IP address is found, all IP addresses are put into the list.

SourceDestination

defines the IP address to be selected and can be SRC, DST or any combination of SRC and|or DST. Ommiting SourceDestination is equivalent to SRC or DST.

inout

defines the interface to be selected and can be IN or OUT.

network

[SourceDestination] NET a.b.c.d m.n.r.s. for IPv4 with m.n.r.s as netmask.

[SourceDestination] NET <net> / num with <net> as a valid IPv4 or IPv6 network and num as maskbits. The number of mask bits must match the appropriate address familiy IPv4 or IPv6. Networks may be abreviated such as 172.16/16 if they are unambiguous.

ICMP

ICMP-TYPE num, ICMP-CODE num with num as a valid icmp type/code respectively. This implies automatically PROTO ICMP

Port

[SourceDestination] PORT [comp] num with num as a valid port number. If comp is omitted, '=' is assumed.
[SourceDestination] PORT IN [<portlist>]
portlist space separated list of individual port numbers

Interface

[inout] IF num with num as an interface number.

Flags

flags tcpflags with tcpflags as a combination of:

A ACK.
S SYN.
F FIN.
R Reset.
P Push.
U Urgent.
X All flags on.

The ordering of the flags is not relevant. Flags not mentioned are treated as don't care. In order to get those flows with only the SYN flag set, use the syntax 'flags S and not flags AFRPU'.

TOS

Type of service: tos value with value 0..255.

Packets

packets [comp] num [scale] to specify the packet count in the netflow record.

Bytes

bytes [comp] num [scale] to specify the byte count in the netflow record.

Packets per second: Calculated value.

pps [comp] num [scale] to specify the pps of the flow.

Duration: Calculated value

duration [comp] num to specify the duration in miliseconds of the flow.

Bits per second: Calculated value.

bps [comp] num [scale] to specify the bps of the flow.

Bytes per packet: Calculated value.

bpp [comp] num [scale] to specify the bpp of the flow.

AS

[SourceDestination] AS num with num as a valid AS number.

scale scaling factor. Maybe k m g. Factor is 1024

comp The following comparators are supported:

=, ==, >, <, EQ, LT, GT . If comp is omitted, '=' is assumed.

EXAMPLES

nfdump -r /and/dir/nfcapd.200407110845 -c 100 'tcp and ( src ip 172.16.17.18 or dst ip 172.16.17.19 )' Dumps the first 100 netflow records which match the given filter:

nfdump -R /and/dir/nfcapd.200407110845:nfcapd.200407110945 'host 192.168.1.2' Dumps all netflow records of host 192.168.1.2 from July 11 08:45 - 09:45

nfdump -M /to/and/dir1:dir2 -R nfcapd.200407110845:nfcapd.200407110945 -S -n 20 Generates the Top 20 statistics from 08:45 to 09:45 from 3 sources

nfdump -r /and/dir/nfcapd.200407110845 -S -n 20 -o extended Generates the Top 20 statistics, extended output format

nfdump -r /and/dir/nfcapd.200407110845 -S -n 20 'in if 5 and bps > 10k' Generates the Top 20 statistics from flows comming from interface 5

nfdump -r /and/dir/nfcapd.200407110845 'inet6 and tcp and ( src port > 1024 and dst port 80 ) Dumps all port 80 IPv6 connections to any web server.

NOTES

Generating the statistics for data files of a few hundred MB is no problem. However be careful if you want to create statistics of several GB of data. This may consume a lot of memory and can take a while. Also, anonymizing IP addresses is time consuming and uses a lot of CPU power, which reduces the number of flows per second. Therefore anonymizing takes place only, when flow records are printed or written to files. Any internal flow processing takes place using the original IP addresses.

SEE ALSO

nfcapd(1), nfprofile(1), nfreplay(1)

BUGS

There is still the famous last bug. Please report them - all the last bugs - back to me.