What is JKFlow.pm?JKFlow is an easy XML configurable Flowscan module for analyzing flowfiles exported with NetFlow from Cisco routers. In JKFlow you can isolate network parts using "directions", using source- and destination subnets/sites or routergroups. Using these directions you can isolate parts of the captured netflow to measure several parameters like total, protocols, services, etc... The design makes JKFlow perfect for branch site WAN-traffic monitoring. This module can serve as a basis of your network monitoring/billing infrastructure.
Example schema 1:
This scheme shows the principe behind JKFlow. We have 3 offices connected with routers over the Internet. Each site is defined using a subnet and contains 2 further subnets: subnet desktops and subnet servers. We want to monitor server related traffic between the servers and to monitor applications between the desktops. JKFlow defines these entities: Sites are defined as a part of the network using (no-)subnets. Directions are defined as sections using source and destination subnets or sites, for specifying flows to monitor. Inside each direction we define services, protocols, applications, tos, total traffic monitoring. Appications are groups of services, like web=http+https and mail=pop+imap+smtp. Also you can use of definesets in directions, to template and reuse services, subnets, applications, tos and total. In every direction we can define different traffic monitoring. Inside every direction we collect RRDTool-data. We can also define scoreboarding based on IP-address or port inside each direction, which are reported in html-files. All the collected RRDTool-data is accessable using a CGI-script JKGrapher, which produces graphs.
Example schema 2:
This scheme shows the new abilities of JKFlow3. You can define sections of routers or router interfaces called routergroups and monitor traffic passing over directions associated with these routergroups. You can even monitor traffic of a specific direction of source and destination subnets over these routergroups. The best of all it that even in this scenario maximum benifit of Net::Patricia subnet lookup speed was maintained.
Is it fast?
Yes!, JKFlow is evolved through many versions and much effort is put make this module both
flexible and fast:
This Flash animation (4 Meg download) shows a demo of the JKGrapher
CGI-interface, showing traffic over 2 interfaces on a router
(located on sourceforge)
Needed skills:You need to know how to install and run FlowScan, located on the Caida website. All configuration is done with a XML-file. If you understand some XML and know the configuration rules, you can configure any monitoring configuration you want. Some little Perl debugging/programming knowledge would be a advantage.
Status:-JKFlow version 3.5.2 is the latest version and includes tuple/time aggregrated scoreboarding, dscp-values.
-Older JKflow 1, 2 are available but not recommended anymore, because of missing features, confusing configuration, lack of speed etc...
-the new SMP version are still a bit buggy, most synchonisation problems.
Prerequisites:JKFlow.pm needs Net::Patricia for fast subnet lookup and XML::Simple for
Legal:JKFlow was developed started from CUFlow and is GPL'ed.
Documentation:Introduction Brochure JKFlow
The Manual of JKFlow (08/09/2005) (recommended reading!):
-NetFlow and flow monitoring issues.
-Design of FlowScan flow reporing architecture.
-Installation of FlowScan in Linux/Solaris.
-Comparisons with different FlowScan reporting modules.
-Argumentation / internals JKFlow design.
-Configuration rules JKFlow 3.5.1.
The internal JKFlow::mylist datastructure layout.
Some example configurations for JKFlow v3.5.1.
Some example configurations for JKFlow v4.0.
This is a powerpoint presentation that I gave on Groep T Leuven.
Screenshots:FlowScan running JKFlow.pm
FlowScan running JKFlow.pm (2)
Daily Windows syncronisation
Daily FTP syncronisation
Hourly Router statistics query
6-hourly Exchange replication
Traffic profile of a subnet
Attack of the Slapper & Code Red worm
Interface monitoring of an inbound FTP-session
Interface monitoring of the same FTP-session outbound
JKGrapher in JKFlow 3.1 produces thin style graphs
Flowscan + JKFlow 3.2.2 on a Multiprocessor Machine
Menu JKGrapher.pl 3.5
The scoreboard directories, + scoreboardfile 3.5
Part of datastructure showing scoreboarding
JKPlotter.pl can overlay tuple line graphs over stacked service graphs
Graphs of JKPlotter showing individual tuples
JKPlotter (absolute) graph of Top-10 In/Outbound tuples
Another JKPlotter overlay graph of tuples over services
JKPlotter.pl has additional tuple selection, the listboxes are sorted by amount of traffic
Comparison JKGrapher/JKPlotter 1
Comparison JKGrapher/JKPlotter 2
Comparison JKGrapher/JKPlotter 3
Comparison JKGrapher/JKPlotter 4
Automatic Install Script (Development, 5/4/2006):-Downloads and installs FlowScan 1.006, flow-tools-0.67, RRDTool 1.2.11 & JKFlow 3.5.2
-Download the script, chmod +x flowscan_install.sh, execute ./flowscan_install.sh
-Tested in Centos 4.2, Fedora 5
-Warning: This script disables SELinux
Centos 4.2 flowscan_install.sh
Fedora 5 flowscan_install.sh
Version 4.0 Alpha (Alpha, 28/02/2006):-Adds DBM::Deep + GD::Grapher support as alternative to RRDTool
-Adds JKPlotter.pl CGI script (URL compatible with JKGrapher.pl)
-Adds top consumer graphing (sorted in listboxes in JKPlotter)
Version 3.5.2 (Stable, 9/02/2006):Fixes:
-Bugfix in combination of routergroups and Autonomous Systems in directions.
-Bugfix in DSCP for VoIP traffic.
-Removed hardcoded Scorekeep and Numkeep attributes.
-Updated directory creation code to only create directories when needed.
-Changed ordering in/out direction in scoreboards.
Version 3.5.1 (Beta, 11/11/2005):
Version 3.4 (Stable, 7/01/2005):
JKFlow2 branch (old):
JKFlow1 branch (very old):
Version 02/06/2003:Added features:
-Improvement in localsubnets, less Net::Patricia->match_integer calls. -Removed unused Multicast-Tos monitoring. Localsubnets are mandatory for all and routers. Migrated countmulticasts into countpackets. The performance should be beter. Avoiding including totals of directions inside selected router/subnet/direction totals.
If you have JKFlow running on your network, I really would like to know
the size of your network, your operating system,CPU,speed and the size/structure
of your configuration file, the number of flows and the time it takes
to process it. Flowfiles are appreciated too. Please send it to jurgen.kobierczynski (at) pandora.be
CVS:Browse the cvs-tree
Post questions / Browse the mailinglist
This project is hosted on sourceforge, Visit the official JKFlow website