Linux Network Monitoring Tools

Something's happening here
What it is, ain't exactly clear ...


With Unix' long tradition of interconnected systems, and its being the first operating system ever to have TCP/IP, with the Open Source community's affinity with standard (networking) protocols, and the programming skills of thousands to build tools to suit their needs, Linux is the ideal platform to run network monitoring and troubleshooting tools. So, if you want to know what's going on on your network, set up Linux, and look around.

set up Linux

... or just use an existing linux system, like this Linux Small Business Server.
Apart from the basic tools (ping, traceroute, ...), network monitoring can be rather CPU-intensive : the monitoring host will capture and process lots of packages, and do lots of computations (moving averages, draw graphs, ...). So you might consider using a dedicated host to do the monitoring. That has a drawback in that you probably want to monitor other hosts' traffic, not necesarilly the trafic from/to/through the system that is doing the monitoring. This complicates the configuration, and might require some changes to your network - e.g. how do you deal with switched networks ?

set up packages

In most cases you can just apt-get the required packages, but sometimes it pays to install the source from the developer's web site, to get to the most recent versions quickly. In order to let apt-get know that a package has been installed this way, you can use 'equivs'. This incorporates the 'build from source' packages into the debian package management, so that other packages can 'depend' on the packages that were not installed with apt-get.

Define Monitoring

Depending on what you want (or need) to know, network monitoring can be a lot of things : do you want to know who's been using up all the bandwith of your internet connection ? Do you want to test connectivity ? Do you need to troubleshoot a client that seems unable to connect to a server ? Are you worried about open ports and possible exploits on networked hosts ? You have no idea who's on your network and what they're doing, but you'd like to find out ?...

The 'Network (Monitoring) Tools' listed here can all contribute - to some extend - to helping you find out what's happening on your network, but as with all data gathering tools, you will have to figure out what you need to know, and how to interprete the data, in order to get some useful information out of it. Otherwise you're just watching pretty pictures, or text scrolling on your screen. Which can be fun ass well.

Set up Network Monitoring in 5 minutes

Need some quick stats ? This mini-howto at Ubuntuforums is just what you need. Set up Network Monitoring in 5 minutes, pvalois, Ubuntu forums.

Basic Tools - Connectivity tests

ping

pingchecks wheter a given host (by name or ip address) is online / reacheable. Further connectivity diagnostics are possible by means of command line options (Linux Journal : an overview of ping).

To ping multiple hosts or a complete address range, you can put the ping command in a loop, eg:

	# ping a class C subnet, 3 pings per host, summary info only

	for ((n=1; n <=254; n++)); do 
		echo;echo
		ping -q -c 3 192.168.1.$n
	done ;
	

To ping multiple hosts, you can also use fping

fping

ping mulptiple hosts or IP ranges (from stdin, inputfile, or range specified in options

	Usage: fping [options] [targets...]
   -a         show targets that are alive
   -A         show targets by address
   -b n       amount of ping data to send, in bytes (default 56)
   -B f       set exponential backoff factor to f
   -c n       count of pings to send to each target (default 1)
   -C n       same as -c, report results in verbose format
   -e         show elapsed time on return packets
   -f file    read list of targets from a file ( - means stdin) (only if no -g specified)
   -g         generate target list (only if no -f specified)
                (specify the start and end IP in the target list, or supply a IP netmask)
                (ex. fping -g 192.168.1.0 192.168.1.255 or fping -g 192.168.1.0/24)
   -i n       interval between sending ping packets (in millisec) (default 25)
   -l         loop sending pings forever
   -m         ping multiple interfaces on target host
   -n         show targets by name (-d is equivalent)
   -p n       interval between ping packets to one target (in millisec)
                (in looping and counting modes, default 1000)
   -q         quiet (don't show per-target/per-ping results)
   -Q n       same as -q, but show summary every n seconds
   -r n       number of retries (default 3)
   -s         print final stats
   -S addr    set source address
   -t n       individual target initial timeout (in millisec) (default 500)
   -u         show targets that are unreachable
   -v         show version
   targets    list of targets to check (if no -f specified)
	

traceroute

traceroute shows the route that an IP package would take to reach a given host. You can also do this by using ping -R (Record Route option), which is faster, but the -R option to ping is ignored by some routers / hosts.

Traceroute can be interesting to test connectivity, establish routes, or map network layouts, but as this can be used to prepare an attack on a network or the hosts in it, firewalls tend to block traceroute packets.

tcptraceroute

tcptraceroute offers traceroute functionality but using tcp packets (to a given port) to circumvent packet filters that block ping and traceroute (udp, icmp) packets. It can also detect and report network address translation (DNAT; masqueraded, NATed destination addresses).

netstat

shows 'sockets', thus network connections. Options to select the type of connections (eg by protocol or address family), and the state of the connections. Interesting if you want to know which hosts your computer is connected to (or: which computers are connecting to your computer ...)

Network usage

The following software is meant to 'look at the LAN' as a whole : which hosts are there, what are they doing, ...

ftp

ftp isn't a network monitor, but can be easily used to get a first impression of network speeds / throughput. ftp a resonable large file from a fast server with good upload speeds, and see how fast you're able to pull it in.

 	...@klix$ ftp -v ftp.belnet.be
	Connected to niue.belnet.be.
	220 ProFTPD 1.3.0a Server (BELNET FTPD Server) [193.190.198.20]
	 (ftp.belnet.be): anonymous
	331 Anonymous login ok, 
	230 Anonymous access granted, restrictions apply.

	Using binary mode to transfer files.

	ftp> cd debian-cd/current/i386/iso-cd
	ftp> get debian-40r0-i386-CD-1.iso
	200 PORT command successful
	150 Opening BINARY mode data connection for debian-40r0-i386-CD-1.iso (679430144 bytes)
	226 Transfer complete.

	679430144 bytes received in 1033.54 secs (642.0 kB/s)
	ftp>

	

ntop

ntop reports network usage, much like the unix top tool reports system resource usage. ntop runs as a daemon and severs webpages at http://yourserver:3000. You can start the daemon with ntop -d; or it will auto-start at system startup. you can also use it as a command line tool. Extensive discussion of arguments and options in the man page. You'll probably want to run it as a daemon and use the web interface to configure and use ntop. A basic setup is really simple : install ntop, run it, and view the results by pointing a browser to port 3000 of the machine where ntop is running. .

	apt-get install ntop
	ntop
	firefox http:/localhost/:3000
	

example output : Network usage during an ftp file transfer from the internet, an indication of download speed :

Actual937.3 Kbps114.5 Pkts/sec
Last Minute0.0 bps0.0 Pkts/sec
Last 5 Minutes0.0 bps0.0 Pkts/sec
Peak937.3 Kbps114.5 Pkts/sec
Average902.6 Kbps110.3 Pkts/sec

And another one ...

Host Domain DataPackets
CurrentAvgPeakCurrentAvgPeak
ftp.belnet.be OS: Solarisbe 4.5 Mbps1.1 Mbps4.5 Mbps570.2 Pkts/sec139.1 Pkts/sec570.2 Pkts/sec

If you're new to network monitoring, ntop in its default configuration will probably show you everything you need to know, and more. Advanced use will require advanced configuration. You can configure ntop from the web interface. Interesting documentation :

Have a look at nagios as welll - see further down this page.

iftop

iftop does for network usage what top(1) does for CPU usage. It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts. Handy for answering the question "why is our ADSL link so slow?". So if you're just looking to quickly investigate and troubleshoot a network link without need for extensive statistics as in ntop, iftop is your tool.

apt-get install iftop
apt-get install iftop

netperf

netperf is a benchmark that can be used to measure the performance of many different types of networking. It provides tests for both unidirecitonal throughput, and end-to-end latency.

Debian keeps a version of nerperf in its 'non-free' section, so add this to the sources list if it isn't their already. Then just apt-get install netperf (netserver get's installed with it).

netperf follows a client-server model : you run netserver on 1 machine, then use netperf on a second machine to send "tests" or commands to the server. The server sends back the testresults for netperf to show on screen. It's a command line tool, and you will have to read the manual to get started with it.

netmeter

netmeter goes a step beyound monitoring a network : it's a tool to test and measure throughput, speed, performance of an IP network - intended to test all "Quality of Service" parameters.

netmeter requires xwindows so you need at least a minimal GUI setup or redirect the GUI to a remote X server.

ipfm

	apt-get installl ipfm
	

ipfm measures the bandwidth usage of every/any host on your network. You need to edit /etc/ipfm.conf to tell it what to monitor and how to report. What follows is an example of ipfm used to find out the individual internet bandwith usage of hosts on a LAN connected to the internet. Network traffic between hosts on the LAN is ignored.

	# Add/Remove following line to toggle enable/disable of this config
	# #DISABLED

	# Global variables

	# analyses configurations

	##### FIRST LOGGING CONFIGURATION #####
	#log our subnet but only with hosts outside the subnet
	LOG 192.168.111.0/255.255.255.0 NOT WITH 192.168.111.0/255.255.255.0

	# path and name for logfile
	FILENAME "/var/log/ipfm/%Y_%d_%m/%H_%M"

	# log every hour at exactly 0:05, 1:05, 2:05 etc.
	DUMP EVERY 5 minute

	# clear statistics each day (at 00:05 UTC)
	CLEAR EVERY 24 hour

	#sort data by host receiving (downloading) most
	SORT IN

	# show hostnames i.s.o. IP addresses
	RESOLVE

	##### SECOND LOGGING CONFIGURATION #####
	# used to specify multiple configurations

	# start a new log file
	NEWLOG
	
	# add 2nd config here
	

ipfm collects data (statistics) in RAM. The DUMP keyword specifies the interval at which to create log files. The log files thus show cumulative results. The CLEAR keyword sets the interval at which the statistics are reset to 0. So the output of the config given here would be:

	kdunix:/var/log/ipfm# ls -R -1
	./2006_12_11:
		15_05
		15_10
		15_15
		15_20
		15_25
		15_30
		15_35
	

and the cumulative result on December 11th, 2006 at 15:35 is in the file /var/lig/ipfm/2006_12_11/15_35 :

	kdunix:/var/log/ipfm# cat /var/log/ipfm/2006_12_11/15_35
	# IPFMv0.11.5 2006/11/12 15:35:00 (local time) -- dump every 0d00:05:00 -- listening on eth0
	# Host                                  In (bytes)    Out (bytes)  Total (bytes)
	192.168.111.7                             28147429         713130       28860559
	kdunix.whitespace.xx                        705583          59989         765572
	192.168.111.12                              201501          34783         236284
	# end of dump 2006/11/12 15:35:00
	

You can use NEWLOG to specify a 2nd (3th, ...) configuration, eg for monthly totals : dump every month and clear statistics after each dump.

	LOG 192.168.111.0/255.255.255.0 NOT WITH 192.168.111.0/255.255.255.0

	#montly stats and logs, logfile named like YEAR_MONTH_DAY
	FILENAME "/var/log/ipfm/%Y_%m_%d"
	DUMP EVERY 30 day
	CLEAR ALWAYS

	# output formatting and options
	SORT IN
	RESOLVE
	

see 'man ipfm.conf' for ipfm options and syntax of the ipfm configuration file. Note that I haven't tested the 'monthly statistics' configuration - I can imagine it takes up a lot of memory or that the 'number of bytes' exceeds the limits of the program so it may crash.

ipfm is started with 'ipfm'. I don't know how to stop it, but at least it can be killed by looking up the process id.

	kill $(pidof ipfm) ;
	

iptraf

iptraf is a console-based network statistics utility for Linux. It gathers a variety of figures such as TCP connection packet and byte counts, interface statistics and activity indicators, TCP/UDP traffic breakdowns, and LAN station packet and byte counts.

cheops

Cheops is an Open Source Network User Interface. It is designed to be the network equivalent of a swiss-army knife, unifying your network utilities. At the time of this writing, the maintainer is looking for a replacement, so development may be slow for a while.

cheops requires a Graphical environment, preferably GNOME and thus is more suitable to run on workstations rather than servers. Produces very nice pictures and can use snmp to gather information about remote systems.

mrtg

apt-get install mrtg mrtg-contrib

mrtg -- Multi Router Traffic Grapher is an snmp client, mainly intended to query routers by snmp, but any device that supports snmp can be queried. It draws pretty pictures ... in a web browser. Again, you'll need to use this with a web server (preferably Apache) to get graphical output.

To effectively use mrtg, you'll need to know a bit about snmp, and you'll have to know how to configure and run mrtg. (Scroll past the (far too many) adds by google).

If you use (Apache) web server to display the output, you may want to make sure it refreshes and doe not show cashed graphs.

Ipac-ng

IPAC-NG is an iptables/ipchains based IP accounting package for Linux. It collects, summarizes, and nicely displays IP accounting data. Its output can be a simple ASCII table, or graph images. Ipchains and iptables are supported. Logs are stored in files, gdbm, or PostgreSQL database.

hunt

hunt is a command line network scanner, and then some. It can detect and watch connections between 2 hosts on the network (other than the computer it is running on), and will even allow you to break into those connections, intercepting the traffic, modify it, then send it on to the receiving host. This can easily lead to IP and MAC address spoofing, man-in -the -middle techniques, sending rogue commands to a host (on behalf of an other), etc.

installation is easy (apt-get install hunt). Then just run hunt : you'll get some sort of text-mode menu. Using and knowing what youre doing is a bit harder. Read the manual.

dsniff

dsniff is a collection of tools for network auditing and penetration testing. This includes tools to facilitate the interception of network traffic which is normally unavailable e.g due to layer-2 switching. From the author's website : "I wrote these tools with honest intentions - to audit my own network, and to demonstrate the insecurity of most network application protocols. Please do not abuse this software."

ethereal (WireShark)

ethereal, now renamed to wireshark, is a packet sniffer and protocol ananlyser. It collects packets from the network, and outputs them on screen so you can actually see hosts talking to each other : connections being initiated and set up or denied, data being exchanged, ... If you need to see every bit passing throug a wire (or even the air waves), from ethernet frames to IP headers and data in payloads, Output can be sorted and filtered by protocol, destination address, source address etc, or organised in conversations (who's talking to who, and what are they saying ?) and you can work with a 'live capture' or with previous captures saved to files.

ngrep

ngrep works like grep (find a string / pattern) in network traffic (payload)

echo server

To test network connectivity on the application level, i.e. to answer a question such as "given the current network design / firewall rules / ...., will a client on this network be able to connect to a server listining on port XXX at address nnn.nnn.nnn.nnn ?" you can use an 'echo server'

monitoring and testing remote hosts

nagios

nagios is the type of monitoring tool that checks of your servers are up and running and available on the network, and notifies you if they're not. It's very advanced, and therefore quite a handful to set up and use. Try ntop first - maybe that is all you need.

To install it "by hand" and configure it is quite a job. I've done a rather extensive write up that goes by the name of the definitive Quickstart Beginners Guide to Nagios in 24 hours for Dummies.

nmap

nmap is the ultimate port scanner / network exploration tool. You can apt-get install nmap, but this might give you an outdated version. With a mixed system and pinning you may be able to get a reasonably new version. But you can also use the most recent (stable) version from insecure.org. It's distributed as source code, so you'll have to compile the program yourself. This should do the trick (replace the .tar.bz2 file name by that of the most recent version and point the wget url to the appropriate download location):

	cd /tmp
	
	apt-get install open-ssl
	wget http://www.insecure.org/nmap/dist/nmap-4.11.tar.bz2

	bzip2 -cd nmap-4.11.tar.bz2 | tar xvf -
	cd nmap-4.11
	./configure
	make
	su root
	make install
	

you can also install the latest version from Debian's 'unstable' branch.

nessus

nessus is a scanner / auditing tool. It can do port scans, determin which services are listening, and even try to apply exploits against these services. It also checks fro a large number of common security holes and offers advice on how to cure them. Some tests require additional plug-ins.

Nessus used to be open source and free. Not anymore.

nessus is a client server application. The server does the work, the client displays it to the user. To install the server, apt-get install nessusd. start the server (nessusd &). Then install the client. You can run the client (nessus) on any other linux/unix-like machine (it will connect to the server), or you can install and run it from the same machine. nessusd will be started at system startup. nessus can be run as a command line tool or as an X application. Refer to man pages. Output can be presented as html for viewing in a browser.

after setup, execute nessus-adduser to create a user with which you'll logon to nessusd.

Then, run nessus (X) or nessus -q (command line, batch mode). You'll have to specify targets, ports, and some other parameters to steer nessus' behaviour.

SATAN

Security Auditing Tool for Analysing Networks. This is a powerful command line tool for searching for vulnerabilities on networked computers.

Wireless Networks

penetration test : password cracking on wireless networks.

web-based management and configuration tools

Switches, routers, network printers, ... these days all have have http interfaces so that you can access a management and configuration page with a web browser. To centralise all these on this one server you could add a page with links to all those on the web server you're already running (see nagios e.a.) so they're all neatly together on this one server, but easily accessible from any workstatin that has a browser.

Alternatively, you could equip your net monitoring Linux system with a minimal GUI and run a browser in it.

Log analysing, monitoring, auditing, intrusion detection and prevention

snare, snort, tripwire, honeyd, ...

All the rest

You can view available packages in aptitude section "net" or on Debian Packages.

interesting tools : iptables, tcpdump, telnet, netcat, nbtscan, hping2, hping3, arp, nstreams, ...

Password tools : john, cain, able, hydra

interesting links

Debian Administrators Network Tools
Collection of network monitoring tools
Network Monitoring With Debian Linux
Basic information on how to monitor networks and network devices, and instructions on how to use ntop and mrtg
And away we spoof
(pdf) document explaining some network monitoring techniques, including IP address spoofing (arp spoof) to capture packets destined to an other host.

Koen Noens
June 2006