JKFlow.pm, the XML-configurable Flowscan module

What is JKFlow.pm?

Example schema 1:

This scheme shows the principe behind JKFlow. We have 3 offices connected with routers over the Internet. Each site is defined using a subnet and contains 2 further subnets: subnet desktops and subnet servers. We want to monitor server related traffic between the servers and to monitor applications between the desktops. JKFlow defines these entities: Sites are defined as a part of the network using (no-)subnets. Directions are defined as sections using source and destination subnets or sites, for specifying flows to monitor. Inside each direction we define services, protocols, applications, tos, total traffic monitoring. Appications are groups of services, like web=http+https and mail=pop+imap+smtp. Also you can use of definesets in directions, to template and reuse services, subnets, applications, tos and total. In every direction we can define different traffic monitoring. Inside every direction we collect RRDTool-data. We can also define scoreboarding based on IP-address or port inside each direction, which are reported in html-files. All the collected RRDTool-data is accessable using a CGI-script JKGrapher, which produces graphs.

Example schema 2:

This scheme shows the new abilities of JKFlow3. You can define sections of routers or router interfaces called routergroups and monitor traffic passing over directions associated with these routergroups. You can even monitor traffic of a specific direction of source and destination subnets over these routergroups. The best of all it that even in this scenario maximum benifit of Net::Patricia subnet lookup speed was maintained.

Is it fast?

Yes!, JKFlow is evolved through many versions and much effort is put make this module both flexible and fast:

It uses C coded Net::Patricia subnet matching module for fast subnet lookups.
It uses Hash-lookups, which are effective. The access-time of a hash-entry is invariant to the amount of entries. Used during counting of multiple protocols/services, this will result in no speed penalty.
It uses a single source/destination inbound+outbound Net::Patricia match per flow. Returning all matching subnets, included and excluded, all matching directions are found via a hash entry. Using more directions won't slow parsing flows much.
It creates for every direction it's own autogenerated evaluation subroutine Every direction can have it's own set of services/protocols/ftp/total/tos to monitor. Apart from it's great flexibility, the several features won't slow parsing, because a unused feature will not be included in the autogenerated code. No other report module use this technique!
Even the wanted subroutine is autogenerated! When you only want to measure overall traffic, only the overall counting function is generated in the wanted subroutine for evaluating flows.
Direction-matching is adapted to best If you create directions based only on source/destination subnets/sites, the code for evaluation of route exporters is not used in the wanted subroutine. When creating directions based on route exporters only, it won't include any Net::Patricia matching code.

Demo:

This Flash animation (4 Meg download) shows a demo of the JKGrapher CGI-interface, showing traffic over 2 interfaces on a router (located on sourceforge)
Flash Demo JKFlow

Needed skills:

Status:

Prerequisites:

Legal:

Documentation:

Screenshots:

Download:

JKFlow3 branch:

Automatic Install Script (Development, 5/4/2006):

Version 4.0 Alpha (Alpha, 28/02/2006):

Version 3.5.2 (Stable, 9/02/2006):

Version 3.5.1 (Beta, 11/11/2005):

Added features:
-DSCP values
-Absolute values in JKGrapher
-Defining custom tuples in scoreboarding
-Added flow-tools 0.67 patch for gcc 3.4
-Added AS-support
Fixes:
-Division by Error bug
-Numkeep incorrect implemented max number of tuples to keep
-Correction in example JKFlow.xml
jkflow-v3.5.1.tgz
multiprocessor jkflow-v3.5.1.smp.tgz

Version 3.4 (Stable, 7/01/2005):

Added features:
-Updating link to latest individual scoreboard
-Aggregated scoreboarding, added <report/> element
-Aggregated scoreboarding possible without individual flowfile scoreboarding
-Added 'other' direction for services not reported by other directions.
-Added sampletime element.
BugFix:
-Warn bug in 3.3.2 fixed.
jkflow-v3.4.tgz
multiprocessor jkflow-v3.4.smp.tgz

JKFlow2 branch (old):

Version 03/11/2003:

Added features:
-Scoreboard host & port reporting
-The xml element <scoreboard/> has 2 new attributes called "hosts" and "ports",
-specifying with the value "1" enables the host and port reporting functionality
-The host & port reports will be written in separate directories.
jkflow2-v03112003.tgz

JKFlow1 branch (very old):

Version 02/06/2003:

If you have JKFlow running on your network, I really would like to know the size of your network, your operating system,CPU,speed and the size/structure of your configuration file, the number of flows and the time it takes to process it. Flowfiles are appreciated too. Please send it to jurgen.kobierczynski (at) pandora.be . Thanks!

CVS:

FAQ: