Linux 2.6.13 Update for Legato NetWorker

David Stes
email: stes@pandora.be

November 17, 2005

Abstract:

In the document ``Connection Tracking for Legato NetWorker on Linux'', we described stateful packet filtering of Legato NetWorker network traffic on Linux kernel version 2.4. This paper describes an update for Linux kernel 2.6.13. Not much has changed, so the technology itself is described in the former paper. The focus here is on the differences between Linux 2.4 and Linux 2.6.

Installing the software

The software described here, was developed on Linux Slackware 10.20 (using the kernel 2.6.13 from the ``extra'' packages).

We used the ``subversion'' version control tool to checkout the latest patch-o-matic-ng sources. We ran the following to apply the necessary patches to the Linux kernel source tree:

root@rpcrouter:~# ./runme extra

When prompted to install the RSH and RPC patches, answer ``yes''.

To compile the sources, rebuild the netfilter modules:

root@rpcrouter:~# make menuconfig

The RPC and RSH modules must be marked as ``M'' (build as modules).

Note that in order to see the RPC module, it is necessary to have ``iptables'' support marked (because it is required for masq/nat). If ``iptables'' is not marked, the RPC module will not be visible as an option in the menuconfig tool.

It is a good idea to check the validity of the kernel build setup, by rebuilding the FTP module, or some other module, that is known to work.

The reason is that the ``runme extra'' step has patched some header files of the netfilter source tree. For example, the header file, /usr/include/linux/netfilter_ipv4/ip_conntrack.h contains a modified definition of a union C type. The union ip_conntrack_proto should contain a RSH field (a field specificically for the RSH module).

The changes in the header file have normally no impact on the data structures of netfilter, but it is a good idea to check that the kernel build.

For example, before patching the kernel sources for the RPC and RSH modules, we run md5sum and after patching the kernel, we rebuild the FTP module, and check that the md5sum is still the same:

md5sum /lib/modules/2.6.13.orig/kernel/net/ipv4/netfilter/ip_conntrack_ftp.ko 
1db6d149713a0dc11b7426b215ad8bdb
md5sum /lib/modules/2.6.13/kernel/net/ipv4/netfilter/ip_conntrack_ftp.ko 
1db6d149713a0dc11b7426b215ad8bdb

Obviously, if this does not work, then something is wrong with the compilation setup (maybe with the ``make menuconfig'' or application of the patch), and if the FTP module does not work any longer, then the RPC and RSH modules will certainly not work either.

Testing the RSH module

Enable the RSH server daemon, in /etc/inetd.conf as follows:

shell   stream  tcp     nowait  root    /usr/sbin/tcpd  in.rshd -h -L

Setup a .rhosts file in your home directory or in the home directory of root, and try to run an rsh session:

# lsof -i tcp:514
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
inetd   2624 root    8u  IPv4   2975       TCP *:shell (LISTEN)
# rsh localhost ls /home

Next, setup the firewall and load the ip_conntrack_rsh module (which will, by default, track connections on TCP port 514:

iptables -P INPUT DROP
iptables -A INPUT -j ACCEPT -p tcp -m state --state NEW -m tcp --dport 514
iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
iptables -A INPUT -m state --state RELATED -j ACCEPT
iptables -A INPUT -j REJECT
modprobe ip_conntrack_rsh

This will result in the following ``dmesg'' output:

ip_tables: (C) 2000-2002 Netfilter core team
ip_conntrack version 2.1 (1023 buckets, 8184 max) - 212 bytes per conntrack
ip_conntrack_rsh: registering helper for port #0: 514/TCP
ip_conntrack_rsh: helper match ip   0.0.0.0:514-0.0.0.0:0
ip_conntrack_rsh: helper match mask 0.0.0.0:64512-0.0.0.0:64512

The RSH session will continue to work (thanks to the connection tracking of the RSH module) and if DEBUG is enabled (DEBUG is a compile time option), the following will be in the ``dmesg'' output:

ip_conntrack_rsh: entered
ip_conntrack_rsh: rsh: find rsh stderr port datalen 5
ip_conntrack_rsh: found port 1022
ip_conntrack_rsh: expect related ip   127.0.0.1:0-127.0.0.1:1022
ip_conntrack_rsh: expect related mask 255.255.255.255:64512-255.255.255.255:65535

The port 1022 is a port that is used by RSH for stderr. This port is dynamically negotiated between RSH server and client.

Testing the RPC module with NFS over TCP

As a NFS client, Linux already supports for a while NFS over TCP. However, as an NFS server, Linux 2.6.13 now also supports NFS over TCP, as well (unlike Linux 2.4).

The result of ``rpcinfo -p'' shows the Linux nfsd server listening on TCP port 2049 for NFS versions 2, 3 and 4 :

   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100011    1   udp    760  rquotad
    100011    2   udp    760  rquotad
    100011    1   tcp    763  rquotad
    100011    2   tcp    763  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   udp   1026  nlockmgr
    100021    3   udp   1026  nlockmgr
    100021    4   udp   1026  nlockmgr
    100021    1   tcp   1025  nlockmgr
    100021    3   tcp   1025  nlockmgr
    100021    4   tcp   1025  nlockmgr
    100005    1   udp    771  mountd
    100005    1   tcp    774  mountd
    100005    2   udp    771  mountd
    100005    2   tcp    774  mountd
    100005    3   udp    771  mountd
    100005    3   tcp    774  mountd
    100024    1   udp    779  status
    100024    1   tcp    782  status

It is possible to mount a filesystem via TCP as follows:

mount -o tcp localhost:/home /mnt

Using ``tcpdump'', it can be seen that this generates TCP connections to the portmapper (sunrpc service, TCP 111), to the mountd daemon (TCP port 774 in this case) and to the NFS server (listening on TCP port 2049).

A simple test to verify that no UDP is being used, is to use a netfilter configuration that drops all UDP packets. As long as the -o tcp option is specified for the NFS mount, this works fine (as there is no UDP traffic being usedwhen using NFS over TCP).

In Linux 2.4, we could already use the RPC module with NFS over UDP. But since Legato NetWorker uses RPC over TCP, the NFS over TCP case, is especially interesting to us. The advantage of the Linux 2.6.13 kernel is obvious : all work that is done on the RPC module for NFS over TCP, applies to Legato NetWorker as well (because it is also an RPC over TCP service).

Now that both the NFS server and NFS client support TCP (in Linux 2.6.13), we use the RPC module for NFS over TCP.

The goal is to track getport requests such as:

tcpdump -s 0 -T rpc -i lo -n
06:33:11.073311 IP 127.0.0.1.0x45d1ecc5 > 127.0.0.1.0x6f: 56 getport 100000.2
06:33:11.073568 IP 127.0.0.1.2049 > 127.0.0.1.1171385541: reply ok 28
06:33:11.073836 IP 127.0.0.1.0x727e37cb > 127.0.0.1.0x6f: 112 getport 100005.1
06:33:11.083009 IP 127.0.0.1.2049 > 127.0.0.1.1920874443: reply ok 24

By accepting new connections to the sunrpc port (port 111), and by tracking ``getport'' requests (by the RPC module), netfilter can dynamically track RPC traffic, by accepting packets that are related to the ``getport'' request.

iptables -P INPUT DROP
iptables -A INPUT -j ACCEPT -p tcp -m state --state NEW -m tcp --dport 111
iptables -A INPUT -j ACCEPT -p udp -m state --state NEW -m udp --dport 111
iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
iptables -A INPUT -m state --state RELATED -j ACCEPT
iptables -A INPUT -m rpc -j ACCEPT
iptables -A INPUT -j REJECT

The output of ``dmesg'' is something like :

ip_conntrack_rpc_udp: registering helper for port #0: 111/UDP
ip_conntrack_rpc_udp: helper match ip   0.0.0.0:0->0.0.0.0:111
ip_conntrack_rpc_udp: helper match mask 0.0.0.0:0->0.0.0.0:65535
ip_conntrack_rpc_tcp: registering helper for port #0: 111/TCP
ip_conntrack_rpc_tcp: helper match ip   0.0.0.0:0->0.0.0.0:111
ip_conntrack_rpc_tcp: helper match mask 0.0.0.0:0->0.0.0.0:65535
ip_conntrack_rpc_tcp: disabling Legato NetWorker support for port 0/TCP
ipt_rpc: registering match [rpc] for;
ipt_rpc:   port 111 (UDP|TCP);

When we mount and unmount a filesystem (NFS over TCP) we observe the following in the output of ``dmesg'' (protocol 6 is TCP) :

ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the initiator. [cont]
ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont]
ip_conntrack_rpc_tcp: RPC packet contains procedure request [100003]. [cont]
ip_conntrack_rpc_tcp: allocated RPC req_p for xid=497089046 proto=6 127.0.0.1:803
ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done]

ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the receiver. [cont]
ip_conntrack_rpc_tcp: port found: 2049
ip_conntrack_rpc_tcp: expect related ip   127.0.0.1:0-127.0.0.1:2049 proto=6
ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:65535 proto=255
ip_conntrack_rpc_tcp: packet evaluated. [expect]

ip_conntrack_rpc_tcp: packet is from the receiver. [cont]
ip_conntrack_rpc_tcp: port found: 774
ip_conntrack_rpc_tcp: expect related ip   127.0.0.1:0-127.0.0.1:774 proto=6
ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:65535 proto=255
ip_conntrack_rpc_tcp: packet evaluated. [expect]

Stress test with Legato nhfsstone

Legato wrote a tool in 1989, called nhfsstone, which nowadays is bundled with many Linux or UNIX operating systems.

The tool (pronounced n-f-s-stone, the ``h'' is silent) is a NFS load generating program, generating an artificial load on an NFS client with a particular (tunable) mix of NFS operations.

The documentation states the following:

Legato nhfsstone is provided with no support and without any obligation on the part of Legato Systems, Inc. to assist in its use, correction, modification or enhancement.

If you would like to receive regular information and bug fixes please send your name, and both your Email and U.S. mail addresses to:

        Legato Systems, Inc.
        Nhfsstone
        260 Sheridan Avenue
        Palo Alto, California  94306

        nhfsstone-request@legato.com or uunet!legato.com!nhfsstone-request

and we will add your name to the nhfsstone mailing list. Comments and bug reports should be sent to:

        nhfsstone@legato.com or uunet!legato.com!nhfsstone

When running nhfsstone, a tcpdump sessions shows lots of RPC traffic:

20:33:20.045554 IP 127.0.0.1.0xdd3fc651 > 127.0.0.1.0x6f: 132 set 100003.3
20:33:20.045613 IP 127.0.0.1.2049 > 127.0.0.1.3695167057: reply ok 236
20:33:20.045801 IP 127.0.0.1.0xde3fc651 > 127.0.0.1.0x6f: 148 getport 100003.3
20:33:20.045954 IP 127.0.0.1.0xdf3fc651 > 127.0.0.1.0x6f: 2200  proc #7
20:33:20.046017 IP 127.0.0.1.2049 > 127.0.0.1.3711944273: reply ok 112
20:33:20.046091 IP 127.0.0.1.0xe03fc651 > 127.0.0.1.0x6f: 132 set 100003.3
20:33:20.046148 IP 127.0.0.1.2049 > 127.0.0.1.3728721489: reply ok 236
20:33:20.046216 IP 127.0.0.1.0xe13fc651 > 127.0.0.1.0x6f: 132 set 100003.3
20:33:20.046560 IP 127.0.0.1.2049 > 127.0.0.1.3762275921: reply ok 112

100003 is the program number (RPC program number) of NFS.

Therefore, this is an interesting test to run with the firewall enabled (with the RPC module loaded). The test can be done either with NFS over TCP or with NFS over UDP.

mount -o tcp localhost:/home /mnt
mkdir /mnt/nhfsstone
cd /mnt/nhfsstone
nhfsstone -v -l 10

Arguments for the modules

There is a new way to configure the modules. On Linux 2.4, the /etc/modules.conf file was used. Now there is /etc/modprobe.conf :

root@rpcrouter:~# cat /etc/modprobe.conf
options ip_conntrack_rsh range=16383 ports=7937
options ip_conntrack_rpc_tcp nsrexec=7937 ports=7938
options ip_conntrack_rpc_udp ports=7938
options ipt_rpc ports=7938

Note that the port 7938 is used as portmapper, and that 7937 is configured as rexec port. The range argument for the rsh module was introduced so we are not limited to the default upper limit of 1023 as for the BSD rexec case (to port 514).

Loading the modules

Load the modules as follows:

# modprobe ip_conntrack_rsh
# modprobe ip_conntrack_rpc_tcp
# modprobe ip_conntrack_rpc_udp
# modprobe ipt_rpc

Check that everything is correctly loaded by inspecting the ``dmesg'' output:

ip_conntrack_rsh: helper match mask 0.0.0.0:49152-0.0.0.0:49152
ip_conntrack_rpc_tcp: registering helper for port #0: 7938/TCP
ip_conntrack_rpc_tcp: helper match ip   0.0.0.0:0->0.0.0.0:7938
ip_conntrack_rpc_tcp: helper match mask 0.0.0.0:0->0.0.0.0:65535
ip_conntrack_rpc_tcp: enabling Legato NetWorker support for port 7937/TCP
ip_conntrack_rpc_udp: registering helper for port #0: 7938/UDP
ip_conntrack_rpc_udp: helper match ip   0.0.0.0:0->0.0.0.0:7938
ip_conntrack_rpc_udp: helper match mask 0.0.0.0:0->0.0.0.0:65535
ipt_rpc: registering match [rpc] for;
ipt_rpc:   port 7938 (UDP|TCP);

Configuring the firewall

The rules are the same as before (see the old paper on iptables and Legato NetWorker).

For example,

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:7937
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:7938
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:7938
ACCEPT     all  --  anywhere             anywhere            state ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            state RELATED
ACCEPT     all  --  anywhere             anywhere            RPCs: nsrd(390103),nsrmmd(390104),nsrindexd(390105),nsrmmdbd(390107),nsrstat(390109),nsrjb(390110),rap(390101),rapserv(390102)

Client initiated backups

Assume that a Legato NetWorker server has registered itself with the portmapper to use the following ports:

sh-3.00# rpcinfo -p | grep nsr
    390113    1   tcp   7937  nsrexec
    390103    2   tcp   7991  nsrd
    390109    2   tcp   7991  nsrstat
    390110    1   tcp   7991  nsrjb
    390103    2   udp   8847  nsrd
    390109    2   udp   8847  nsrstat
    390110    1   udp   8847  nsrjb
    390107    5   tcp   8811  nsrmmdbd
    390107    6   tcp   8811  nsrmmdbd
    390105    5   tcp   9364  nsrindexd
    390105    6   tcp   9364  nsrindexd
    390104  105   tcp   8972  nsrmmd
    390104  205   tcp   9761  nsrmmd

In this case, we would expect a client initiated backup (``save'') to contact the Legato master daemon and then subsequently send data to a media multixplor daemon. Indeed, when running the RPC module in debug mode:

ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the initiator. [cont]
ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont]
ip_conntrack_rpc_tcp: RPC packet contains procedure request [390103]. [cont]
ip_conntrack_rpc_tcp: allocated RPC req_p for xid=4056711746 proto=6 127.0.0.1:1
3701
ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the receiver. [cont]
ip_conntrack_rpc_tcp: port found: 7991
ip_conntrack_rpc_tcp: expect related ip   127.0.0.1:0-127.0.0.1:7991 proto=6
ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:6553
5 proto=255
ip_conntrack_rpc_tcp: packet evaluated. [expect]

ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the initiator. [cont]
ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont]
ip_conntrack_rpc_tcp: RPC packet contains procedure request [390104]. [cont]
ip_conntrack_rpc_tcp: allocated RPC req_p for xid=4020404802 proto=6 127.0.0.1:1
5208
ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the receiver. [cont]
ip_conntrack_rpc_tcp: port found: 9761
ip_conntrack_rpc_tcp: expect related ip   127.0.0.1:0-127.0.0.1:9761 proto=6
ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:6553
5 proto=255

ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the initiator. [cont]
ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont]
ip_conntrack_rpc_tcp: RPC packet contains procedure request [390105]. [cont]
ip_conntrack_rpc_tcp: allocated RPC req_p for xid=3162931778 proto=6 127.0.0.1:1
2280
ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done]

ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the receiver. [cont]
ip_conntrack_rpc_tcp: port found: 9364
ip_conntrack_rpc_tcp: expect related ip   127.0.0.1:0-127.0.0.1:9364 proto=6
ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:6553
5 proto=255
ip_conntrack_rpc_tcp: packet evaluated. [expect]

ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the initiator. [cont]
ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont]
ip_conntrack_rpc_tcp: RPC packet contains procedure request [390113]. [cont]
ip_conntrack_rpc_tcp: allocated RPC req_p for xid=1869644354 proto=6 127.0.0.1:1
2070
ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done]

Scheduled backups

When using ``savegrp'', the server makes a connection to the client to run such commands as ``savefs'' and ``save'', in order to do the backup.

In this case, Legato NetWorker will make outbound connections to the port 7937 and inbound connections for redirecting stderr to a dynamically negotiated port :

22:42:06.026788 IP 127.0.0.1.18182 > 127.0.0.1.7937: . ack 210 win 8192 <nop,nop
,timestamp 1200683 1200683>
22:42:06.027168 IP 127.0.0.1.14608 > 127.0.0.1.9660: S 454289648:454289648(0) wi
n 32767 <mss 16396,sackOK,timestamp 1200683 0,nop,wscale 2>
22:42:06.027220 IP 127.0.0.1.9660 > 127.0.0.1.14608: S 459034062:459034062(0) ac
k 454289649 win 32767 <mss 16396,sackOK,timestamp 1200683 1200683,nop,wscale 2>
22:42:06.027258 IP 127.0.0.1.14608 > 127.0.0.1.9660: . ack 1 win 8192 <nop,nop,t
imestamp 1200683 1200683>

The RSH module is able to track those ports :

Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: entered
Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: rsh: find rsh stderr port dat
alen 5
Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: found port 9660
Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: expect related ip   127.0.0.1
:0-127.0.0.1:9660
Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: expect related mask 255.255.2
55.255:49152-255.255.255.255:65535

2


David Stes
2005-11-27