David Stes
email: stes@pandora.be
November 17, 2005
The software described here, was developed on Linux Slackware 10.20 (using the kernel 2.6.13 from the ``extra'' packages).
We used the ``subversion'' version control tool to checkout the latest patch-o-matic-ng sources. We ran the following to apply the necessary patches to the Linux kernel source tree:
root@rpcrouter:~# ./runme extra
When prompted to install the RSH and RPC patches, answer ``yes''.
To compile the sources, rebuild the netfilter modules:
root@rpcrouter:~# make menuconfig
The RPC and RSH modules must be marked as ``M'' (build as modules).
Note that in order to see the RPC module, it is necessary to have ``iptables'' support marked (because it is required for masq/nat). If ``iptables'' is not marked, the RPC module will not be visible as an option in the menuconfig tool.
It is a good idea to check the validity of the kernel build setup, by rebuilding the FTP module, or some other module, that is known to work.
The reason is that the ``runme extra'' step has patched some header files of the netfilter source tree. For example, the header file, /usr/include/linux/netfilter_ipv4/ip_conntrack.h contains a modified definition of a union C type. The union ip_conntrack_proto should contain a RSH field (a field specificically for the RSH module).
The changes in the header file have normally no impact on the data structures of netfilter, but it is a good idea to check that the kernel build.
For example, before patching the kernel sources for the RPC and RSH modules, we run md5sum and after patching the kernel, we rebuild the FTP module, and check that the md5sum is still the same:
md5sum /lib/modules/2.6.13.orig/kernel/net/ipv4/netfilter/ip_conntrack_ftp.ko 1db6d149713a0dc11b7426b215ad8bdb md5sum /lib/modules/2.6.13/kernel/net/ipv4/netfilter/ip_conntrack_ftp.ko 1db6d149713a0dc11b7426b215ad8bdb
Obviously, if this does not work, then something is wrong with the compilation setup (maybe with the ``make menuconfig'' or application of the patch), and if the FTP module does not work any longer, then the RPC and RSH modules will certainly not work either.
Enable the RSH server daemon, in /etc/inetd.conf as follows:
shell stream tcp nowait root /usr/sbin/tcpd in.rshd -h -L
Setup a .rhosts file in your home directory or in the home directory of root, and try to run an rsh session:
# lsof -i tcp:514 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME inetd 2624 root 8u IPv4 2975 TCP *:shell (LISTEN) # rsh localhost ls /home
Next, setup the firewall and load the ip_conntrack_rsh module (which will, by default, track connections on TCP port 514:
iptables -P INPUT DROP iptables -A INPUT -j ACCEPT -p tcp -m state --state NEW -m tcp --dport 514 iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT iptables -A INPUT -m state --state RELATED -j ACCEPT iptables -A INPUT -j REJECT modprobe ip_conntrack_rsh
This will result in the following ``dmesg'' output:
ip_tables: (C) 2000-2002 Netfilter core team ip_conntrack version 2.1 (1023 buckets, 8184 max) - 212 bytes per conntrack ip_conntrack_rsh: registering helper for port #0: 514/TCP ip_conntrack_rsh: helper match ip 0.0.0.0:514-0.0.0.0:0 ip_conntrack_rsh: helper match mask 0.0.0.0:64512-0.0.0.0:64512
The RSH session will continue to work (thanks to the connection tracking of the RSH module) and if DEBUG is enabled (DEBUG is a compile time option), the following will be in the ``dmesg'' output:
ip_conntrack_rsh: entered ip_conntrack_rsh: rsh: find rsh stderr port datalen 5 ip_conntrack_rsh: found port 1022 ip_conntrack_rsh: expect related ip 127.0.0.1:0-127.0.0.1:1022 ip_conntrack_rsh: expect related mask 255.255.255.255:64512-255.255.255.255:65535
The port 1022 is a port that is used by RSH for stderr. This port is dynamically negotiated between RSH server and client.
As a NFS client, Linux already supports for a while NFS over TCP. However, as an NFS server, Linux 2.6.13 now also supports NFS over TCP, as well (unlike Linux 2.4).
The result of ``rpcinfo -p'' shows the Linux nfsd server listening on TCP port 2049 for NFS versions 2, 3 and 4 :
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100011 1 udp 760 rquotad
100011 2 udp 760 rquotad
100011 1 tcp 763 rquotad
100011 2 tcp 763 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 udp 1026 nlockmgr
100021 3 udp 1026 nlockmgr
100021 4 udp 1026 nlockmgr
100021 1 tcp 1025 nlockmgr
100021 3 tcp 1025 nlockmgr
100021 4 tcp 1025 nlockmgr
100005 1 udp 771 mountd
100005 1 tcp 774 mountd
100005 2 udp 771 mountd
100005 2 tcp 774 mountd
100005 3 udp 771 mountd
100005 3 tcp 774 mountd
100024 1 udp 779 status
100024 1 tcp 782 status
It is possible to mount a filesystem via TCP as follows:
mount -o tcp localhost:/home /mnt
Using ``tcpdump'', it can be seen that this generates TCP connections to the portmapper (sunrpc service, TCP 111), to the mountd daemon (TCP port 774 in this case) and to the NFS server (listening on TCP port 2049).
A simple test to verify that no UDP is being used, is to use a netfilter configuration that drops all UDP packets. As long as the -o tcp option is specified for the NFS mount, this works fine (as there is no UDP traffic being usedwhen using NFS over TCP).
In Linux 2.4, we could already use the RPC module with NFS over UDP. But since Legato NetWorker uses RPC over TCP, the NFS over TCP case, is especially interesting to us. The advantage of the Linux 2.6.13 kernel is obvious : all work that is done on the RPC module for NFS over TCP, applies to Legato NetWorker as well (because it is also an RPC over TCP service).
Now that both the NFS server and NFS client support TCP (in Linux 2.6.13), we use the RPC module for NFS over TCP.
The goal is to track getport requests such as:
tcpdump -s 0 -T rpc -i lo -n 06:33:11.073311 IP 127.0.0.1.0x45d1ecc5 > 127.0.0.1.0x6f: 56 getport 100000.2 06:33:11.073568 IP 127.0.0.1.2049 > 127.0.0.1.1171385541: reply ok 28 06:33:11.073836 IP 127.0.0.1.0x727e37cb > 127.0.0.1.0x6f: 112 getport 100005.1 06:33:11.083009 IP 127.0.0.1.2049 > 127.0.0.1.1920874443: reply ok 24
By accepting new connections to the sunrpc port (port 111), and by tracking ``getport'' requests (by the RPC module), netfilter can dynamically track RPC traffic, by accepting packets that are related to the ``getport'' request.
iptables -P INPUT DROP iptables -A INPUT -j ACCEPT -p tcp -m state --state NEW -m tcp --dport 111 iptables -A INPUT -j ACCEPT -p udp -m state --state NEW -m udp --dport 111 iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT iptables -A INPUT -m state --state RELATED -j ACCEPT iptables -A INPUT -m rpc -j ACCEPT iptables -A INPUT -j REJECT
The output of ``dmesg'' is something like :
ip_conntrack_rpc_udp: registering helper for port #0: 111/UDP ip_conntrack_rpc_udp: helper match ip 0.0.0.0:0->0.0.0.0:111 ip_conntrack_rpc_udp: helper match mask 0.0.0.0:0->0.0.0.0:65535 ip_conntrack_rpc_tcp: registering helper for port #0: 111/TCP ip_conntrack_rpc_tcp: helper match ip 0.0.0.0:0->0.0.0.0:111 ip_conntrack_rpc_tcp: helper match mask 0.0.0.0:0->0.0.0.0:65535 ip_conntrack_rpc_tcp: disabling Legato NetWorker support for port 0/TCP ipt_rpc: registering match [rpc] for; ipt_rpc: port 111 (UDP|TCP);
When we mount and unmount a filesystem (NFS over TCP) we observe the following in the output of ``dmesg'' (protocol 6 is TCP) :
ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the initiator. [cont] ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont] ip_conntrack_rpc_tcp: RPC packet contains procedure request [100003]. [cont] ip_conntrack_rpc_tcp: allocated RPC req_p for xid=497089046 proto=6 127.0.0.1:803 ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the receiver. [cont] ip_conntrack_rpc_tcp: port found: 2049 ip_conntrack_rpc_tcp: expect related ip 127.0.0.1:0-127.0.0.1:2049 proto=6 ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:65535 proto=255 ip_conntrack_rpc_tcp: packet evaluated. [expect] ip_conntrack_rpc_tcp: packet is from the receiver. [cont] ip_conntrack_rpc_tcp: port found: 774 ip_conntrack_rpc_tcp: expect related ip 127.0.0.1:0-127.0.0.1:774 proto=6 ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:65535 proto=255 ip_conntrack_rpc_tcp: packet evaluated. [expect]
Legato wrote a tool in 1989, called nhfsstone, which nowadays is bundled with many Linux or UNIX operating systems.
The tool (pronounced n-f-s-stone, the ``h'' is silent) is a NFS load generating program, generating an artificial load on an NFS client with a particular (tunable) mix of NFS operations.
The documentation states the following:
Legato nhfsstone is provided with no support and without any obligation on the part of Legato Systems, Inc. to assist in its use, correction, modification or enhancement.
If you would like to receive regular information and bug fixes please send your name, and both your Email and U.S. mail addresses to:
Legato Systems, Inc.
Nhfsstone
260 Sheridan Avenue
Palo Alto, California 94306
nhfsstone-request@legato.com or uunet!legato.com!nhfsstone-request
and we will add your name to the nhfsstone mailing list. Comments and bug reports should be sent to:
nhfsstone@legato.com or uunet!legato.com!nhfsstone
When running nhfsstone, a tcpdump sessions shows lots of RPC traffic:
20:33:20.045554 IP 127.0.0.1.0xdd3fc651 > 127.0.0.1.0x6f: 132 set 100003.3 20:33:20.045613 IP 127.0.0.1.2049 > 127.0.0.1.3695167057: reply ok 236 20:33:20.045801 IP 127.0.0.1.0xde3fc651 > 127.0.0.1.0x6f: 148 getport 100003.3 20:33:20.045954 IP 127.0.0.1.0xdf3fc651 > 127.0.0.1.0x6f: 2200 proc #7 20:33:20.046017 IP 127.0.0.1.2049 > 127.0.0.1.3711944273: reply ok 112 20:33:20.046091 IP 127.0.0.1.0xe03fc651 > 127.0.0.1.0x6f: 132 set 100003.3 20:33:20.046148 IP 127.0.0.1.2049 > 127.0.0.1.3728721489: reply ok 236 20:33:20.046216 IP 127.0.0.1.0xe13fc651 > 127.0.0.1.0x6f: 132 set 100003.3 20:33:20.046560 IP 127.0.0.1.2049 > 127.0.0.1.3762275921: reply ok 112
100003 is the program number (RPC program number) of NFS.
Therefore, this is an interesting test to run with the firewall enabled (with the RPC module loaded). The test can be done either with NFS over TCP or with NFS over UDP.
mount -o tcp localhost:/home /mnt mkdir /mnt/nhfsstone cd /mnt/nhfsstone nhfsstone -v -l 10
There is a new way to configure the modules. On Linux 2.4, the /etc/modules.conf file was used. Now there is /etc/modprobe.conf :
root@rpcrouter:~# cat /etc/modprobe.conf options ip_conntrack_rsh range=16383 ports=7937 options ip_conntrack_rpc_tcp nsrexec=7937 ports=7938 options ip_conntrack_rpc_udp ports=7938 options ipt_rpc ports=7938
Note that the port 7938 is used as portmapper, and that 7937 is configured as rexec port. The range argument for the rsh module was introduced so we are not limited to the default upper limit of 1023 as for the BSD rexec case (to port 514).
Load the modules as follows:
# modprobe ip_conntrack_rsh # modprobe ip_conntrack_rpc_tcp # modprobe ip_conntrack_rpc_udp # modprobe ipt_rpc
Check that everything is correctly loaded by inspecting the ``dmesg'' output:
ip_conntrack_rsh: helper match mask 0.0.0.0:49152-0.0.0.0:49152 ip_conntrack_rpc_tcp: registering helper for port #0: 7938/TCP ip_conntrack_rpc_tcp: helper match ip 0.0.0.0:0->0.0.0.0:7938 ip_conntrack_rpc_tcp: helper match mask 0.0.0.0:0->0.0.0.0:65535 ip_conntrack_rpc_tcp: enabling Legato NetWorker support for port 7937/TCP ip_conntrack_rpc_udp: registering helper for port #0: 7938/UDP ip_conntrack_rpc_udp: helper match ip 0.0.0.0:0->0.0.0.0:7938 ip_conntrack_rpc_udp: helper match mask 0.0.0.0:0->0.0.0.0:65535 ipt_rpc: registering match [rpc] for; ipt_rpc: port 7938 (UDP|TCP);
The rules are the same as before (see the old paper on iptables and Legato NetWorker).
For example,
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy DROP) target prot opt source destination ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:7937 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:7938 ACCEPT udp -- anywhere anywhere state NEW udp dpt:7938 ACCEPT all -- anywhere anywhere state ESTABLISHED ACCEPT all -- anywhere anywhere state RELATED ACCEPT all -- anywhere anywhere RPCs: nsrd(390103),nsrmmd(390104),nsrindexd(390105),nsrmmdbd(390107),nsrstat(390109),nsrjb(390110),rap(390101),rapserv(390102)
Assume that a Legato NetWorker server has registered itself with the portmapper to use the following ports:
sh-3.00# rpcinfo -p | grep nsr
390113 1 tcp 7937 nsrexec
390103 2 tcp 7991 nsrd
390109 2 tcp 7991 nsrstat
390110 1 tcp 7991 nsrjb
390103 2 udp 8847 nsrd
390109 2 udp 8847 nsrstat
390110 1 udp 8847 nsrjb
390107 5 tcp 8811 nsrmmdbd
390107 6 tcp 8811 nsrmmdbd
390105 5 tcp 9364 nsrindexd
390105 6 tcp 9364 nsrindexd
390104 105 tcp 8972 nsrmmd
390104 205 tcp 9761 nsrmmd
In this case, we would expect a client initiated backup (``save'') to contact the Legato master daemon and then subsequently send data to a media multixplor daemon. Indeed, when running the RPC module in debug mode:
ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the initiator. [cont] ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont] ip_conntrack_rpc_tcp: RPC packet contains procedure request [390103]. [cont] ip_conntrack_rpc_tcp: allocated RPC req_p for xid=4056711746 proto=6 127.0.0.1:1 3701 ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the receiver. [cont] ip_conntrack_rpc_tcp: port found: 7991 ip_conntrack_rpc_tcp: expect related ip 127.0.0.1:0-127.0.0.1:7991 proto=6 ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:6553 5 proto=255 ip_conntrack_rpc_tcp: packet evaluated. [expect] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the initiator. [cont] ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont] ip_conntrack_rpc_tcp: RPC packet contains procedure request [390104]. [cont] ip_conntrack_rpc_tcp: allocated RPC req_p for xid=4020404802 proto=6 127.0.0.1:1 5208 ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the receiver. [cont] ip_conntrack_rpc_tcp: port found: 9761 ip_conntrack_rpc_tcp: expect related ip 127.0.0.1:0-127.0.0.1:9761 proto=6 ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:6553 5 proto=255 ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the initiator. [cont] ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont] ip_conntrack_rpc_tcp: RPC packet contains procedure request [390105]. [cont] ip_conntrack_rpc_tcp: allocated RPC req_p for xid=3162931778 proto=6 127.0.0.1:1 2280 ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the receiver. [cont] ip_conntrack_rpc_tcp: port found: 9364 ip_conntrack_rpc_tcp: expect related ip 127.0.0.1:0-127.0.0.1:9364 proto=6 ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:6553 5 proto=255 ip_conntrack_rpc_tcp: packet evaluated. [expect] ip_conntrack_rpc_tcp: new packet to evaluate .. ip_conntrack_rpc_tcp: packet is from the initiator. [cont] ip_conntrack_rpc_tcp: RPC packet contains a "get" requestor. [cont] ip_conntrack_rpc_tcp: RPC packet contains procedure request [390113]. [cont] ip_conntrack_rpc_tcp: allocated RPC req_p for xid=1869644354 proto=6 127.0.0.1:1 2070 ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done]
When using ``savegrp'', the server makes a connection to the client to run such commands as ``savefs'' and ``save'', in order to do the backup.
In this case, Legato NetWorker will make outbound connections to the port 7937 and inbound connections for redirecting stderr to a dynamically negotiated port :
22:42:06.026788 IP 127.0.0.1.18182 > 127.0.0.1.7937: . ack 210 win 8192 <nop,nop ,timestamp 1200683 1200683> 22:42:06.027168 IP 127.0.0.1.14608 > 127.0.0.1.9660: S 454289648:454289648(0) wi n 32767 <mss 16396,sackOK,timestamp 1200683 0,nop,wscale 2> 22:42:06.027220 IP 127.0.0.1.9660 > 127.0.0.1.14608: S 459034062:459034062(0) ac k 454289649 win 32767 <mss 16396,sackOK,timestamp 1200683 1200683,nop,wscale 2> 22:42:06.027258 IP 127.0.0.1.14608 > 127.0.0.1.9660: . ack 1 win 8192 <nop,nop,t imestamp 1200683 1200683>
The RSH module is able to track those ports :
Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: entered Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: rsh: find rsh stderr port dat alen 5 Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: found port 9660 Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: expect related ip 127.0.0.1 :0-127.0.0.1:9660 Nov 20 22:42:06 darkstar kernel: ip_conntrack_rsh: expect related mask 255.255.2 55.255:49152-255.255.255.255:65535
2