Connection Tracking for Legato NetWorker 7.3

David Stes
email: stes@pandora.be

January 1, 2006

Abstract:

Legato NetWorker 7.3 introduces some new RPC services. In this document, we describe how connection tracking in the style of the previous documents ``Connection Tracking for Legato NetWorker on Linux'' and ``Linux 2.6 Update for Legato NetWorker'' works with Legato NetWorker 7.3.

Installing the software

The software described here, was developed on Linux Slackware 10.20 (using the kernel 2.6.14.4, not the 2.6.13 kernel from the ``extra'' packages). The kernel 2.6.13 is also fine, but we used the more recent 2.6.14.4 kernel from the Linux kernel archives instead.

We used the ``subversion'' version control tool to checkout the latest patch-o-matic-ng sources. See our previous text, ``Linux 2.6 Update for Legato NetWorker'' for more details.

New RPC services

After installing Legato NetWorker 7.3, the /etc/rpc file contains new services, such as:

nsrlcpd         390429
nsrmmgd         390430
nsrjobd         390433

The first service, nsrlcpd, is the new NetWorker library control program daemon. The second service, nsrmmgd, is the new NetWorker media management daemon. And finally, nsrjobd, is the service of the new job monitoring daemon.

If you are using PowerSnap, you may also have some services such as ``nsrpsd'' (powersnap daemon) for that functionality (but this is not specific to Legato NetWorker 7.3).

And of course, the usual NetWorker services, such as ``nsrd'', ``nsrmmd'', ``nsrindexd'', ``nsrmmdbd'' and ``nsrexec'' are also there.

Installing the firewall

As described in the paper ``Linux 2.6 Update for Legato NetWorker'', the following /etc/modprobe.conf file is used:

options ip_conntrack_rsh range=16383 ports=7937
options ip_conntrack_rpc_tcp nsrexec=7937 ports=7938
options ip_conntrack_rpc_udp ports=7938
options ipt_rpc ports=7938

The RPC and RSH modules are then loaded as follows:

# modprobe ip_conntrack_rsh
# modprobe ip_conntrack_rpc_tcp
# modprobe ip_conntrack_rpc_udp
# modprobe ipt_rpc

After starting the Legato NetWorker software, the ``rpcinfo'' output is as follows:

    390436    1   tcp   9773
    390435    1   tcp   9867
    390113    1   tcp   7937  nsrexecd
    390103    2   tcp   9266  nsrd
    390109    2   tcp   9266  nsrstat
    390110    1   tcp   9266  nsrjbd
    390120    1   tcp   9266
    390109    2   udp   8765  nsrstat
    390120    1   udp   8765
    390107    5   tcp   9778  nsrmmdbd
    390107    6   tcp   9778  nsrmmdbd
    390433    1   tcp   8149  nsrjobd
    390105    5   tcp   8449  nsrindexd
    390105    6   tcp   8449  nsrindexd
    390104  105   tcp   8041  nsrmmd
    390104  205   tcp   9906  nsrmmd

The Legato NetWorker configuration in the above consists of one adv_file device (there are two ``nsrmmd'' registered).

bash-3.00# ps -ef | grep nsr
root      9873     1  0 17:52 ?        00:00:00 nsrexecd
root      9881     1  0 17:52 ?        00:00:00 nsrd
root      9886  9881  0 17:52 ?        00:00:00 /usr/sbin/nsrmmdbd
root      9890  9881  0 17:52 ?        00:00:00 /usr/sbin/nsrindexd
root      9898  9881  0 17:52 ?        00:00:00 /usr/sbin/nsrjobd
root      9978  9881  0 18:03 ?        00:00:00 /usr/sbin/nsrmmd -n 1
root     10039  9881  0 18:05 ?        00:00:00 /usr/sbin/nsrmmd -n 2
root     10395  4320  0 19:04 pts/3    00:00:00 grep nsr

The trick that Legato uses in these cases, when there are two nsrmmd processes, is to register multiple instances of ``nsrmmd'' by multiplying the RPC version by 100 and adding the base version. This trick is used by ``nsrmmd'' as well as ``nsrlcpd''.

Apparently, there are two additional (new, unnamed) services 390436 and 390435 being used. It can be seen in the previous papers, on Linux connection tracking with older Legato NetWorker versions, that these RPC ports were not registered, before 7.3. The Linux connection tracking is tracking RPC traffic to those RPC services, as can be seen from the ``dmesg'' output:

ip_conntrack_rpc_tcp: allocated RPC req_p for xid=2798501186 proto=6 127.0.0.1:1
261
ip_conntrack_rpc_tcp: allocated RPC request for protocol 6. [done]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet has no data (may still be handshaking). [skip]
ip_conntrack_rpc_tcp: new packet to evaluate ..
ip_conntrack_rpc_tcp: packet is from the receiver. [cont]
ip_conntrack_rpc_tcp: port found: 9773
ip_conntrack_rpc_tcp: expect related ip   127.0.0.1:0-127.0.0.1:9773 proto=6
ip_conntrack_rpc_tcp: expect related mask 255.255.255.255:0-255.255.255.255:6553
5 proto=255
ip_conntrack_rpc_tcp: packet evaluated. [expect]

There is also UDP RPC traffic going on over the loopback interface:

ip_conntrack_rpc_udp: RPC packet contains procedure request [390109]. [cont]
ip_conntrack_rpc_udp: allocated RPC req_p for xid=135705156 proto=17 127.0.0.1:1
076
ip_conntrack_rpc_udp: allocated RPC request for protocol 17. [done]
ip_conntrack_rpc_udp: new packet to evaluate ..
ip_conntrack_rpc_udp: packet is from the initiator. [cont]
ip_conntrack_rpc_udp: RPC packet contains a "get" requestor. [cont]
ip_conntrack_rpc_udp: RPC packet contains procedure request [390109]. [cont]
ip_conntrack_rpc_udp: allocated RPC req_p for xid=135705156 proto=17 127.0.0.1:1
076
ip_conntrack_rpc_udp: allocated RPC request for protocol 17. [done]
ip_conntrack_rpc_udp: new packet to evaluate ..
ip_conntrack_rpc_udp: packet is from the receiver. [cont]
ip_conntrack_rpc_udp: port found: 8765
ip_conntrack_rpc_udp: expect related ip   127.0.0.1:0-127.0.0.1:8765 proto=17
ip_conntrack_rpc_udp: expect related mask 255.255.255.255:0-255.255.255.255:6553
5 proto=255

Client initiated backup over the firewall

There are two ways to activate the RPC matching: either using Lima's match any RPC service, or by Ian Latter's specific RPC filtering.

For example, using RPC service specific filtering,

iptables -P INPUT DROP
iptables -A INPUT -j ACCEPT -p tcp -m state --state NEW -m tcp --dport 7937
iptables -A INPUT -j ACCEPT -p tcp -m state --state NEW -m tcp --dport 7938
iptables -A INPUT -j ACCEPT -p udp -m state --state NEW -m udp --dport 7938
iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
iptables -A INPUT -m state --state RELATED -j ACCEPT
iptables -A INPUT -m rpc --rpcs nsrd,nsrexec,nsrmmd,nsrindexd,nsrmmdbd,nsrstat,n
srjobd,nsrlcpd,nsrmmgd,390436,390435,390120 -j ACCEPT
iptables -A INPUT -j REJECT

bash-3.00# iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:7937 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:7938 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:7938 
ACCEPT     all  --  anywhere             anywhere            state ESTABLISHED 
ACCEPT     all  --  anywhere             anywhere            state RELATED 
ACCEPT     all  --  anywhere             anywhere            RPCs: nsrd(390103),nsrexecd(390113),nsrmmd(390104),nsrindexd(390105),nsrmmdbd(390107),nsrstat(390109),nsrjobd(390433),nsrlcpd(390429),nsrmmgd(390430),unknown(390436),unknown(390435),unknown(390120) 
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

A client initiated backup over the firewall is started, using ``save'' as in:

bash-3.00#  save /etc/passwd
save: Using darkstar.example.net as server
/etc/passwd
/etc/
/

save: /etc/passwd  6 KB 00:00:05      3 files

We can see after the ``save'' that some of the related and established connections are seen in the statistics of iptables:

bash-3.00# iptables -L -v
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2   120 ACCEPT     tcp  --  any    any     anywhere             anywhere            state NEW tcp dpt:7937 
   22  1320 ACCEPT     tcp  --  any    any     anywhere             anywhere            state NEW tcp dpt:7938 
    1    84 ACCEPT     udp  --  any    any     anywhere             anywhere            state NEW udp dpt:7938 
 1431  192K ACCEPT     all  --  any    any     anywhere             anywhere            state ESTABLISHED 
   23  1424 ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED

Also in the ``dmesg'' output (which is a circular buffer, truncated), we see RPC matching such as:

bash-3.00# dmesg | grep port
ip_conntrack_rpc_tcp: port found: 8449
ip_conntrack_rpc_tcp: port found: 9773
ip_conntrack_rpc_tcp: port found: 9773
ip_conntrack_rpc_tcp: port found: 9773
ip_conntrack_rpc_udp: port found: 8765

Scheduled backups over the firewall

With scheduled backups, the server connects to the client to launch such commands as ``savefs'' and ``save'' :

bash-3.00# savegrp -v -I  
darkstar.example.net:/etc/passwd          level=full
01/01/06 19:17:37 savegrp: Group will not limit job parallelism
01/01/06 19:17:37 savegrp: darkstar.example.net:probe                    started
savefs -s darkstar.example.net -c darkstar.example.net -g Default -p -l full -R -v -F /etc/passwd 
darkstar.example.net:/etc/passwd   level=full, dn=0, mx=1, vers=ssbrowse, p=4
* darkstar.example.net:Probe savefs darkstar.example.net: succeeded.
01/01/06 19:17:40 savegrp: darkstar.example.net:probe succeeded.
01/01/06 19:17:40 savegrp: darkstar.example.net:/etc/passwd              started
save -s darkstar.example.net -g Default -LL -m darkstar.example.net -l full -W 78 -N /etc/passwd /etc/passwd

When inspecting the packets with ``tcpdump'', it can be seen that a connection is initiated to TCP port 3785, although that this port is not registered with the portmapper (see output of rpcinfo above) :

tcpdump -n -s 0 -X -i lo > /tmp/probe

19:37:23.667120 IP 127.0.0.1.3786 > 127.0.0.1.3785: S 2672338085:2672338085(0) w
in 32767 <mss 16396,sackOK,timestamp 4969159 0,nop,wscale 2>
19:37:23.667176 IP 127.0.0.1.3785 > 127.0.0.1.3786: S 2665530445:2665530445(0) a
ck 2672338086 win 32767 <mss 16396,sackOK,timestamp 4969159 4969159,nop,wscale 2

or, after a reboot of our test machine,

21:24:40.784288 IP 127.0.0.1.1709 > 127.0.0.1.1708: S 3961496765:3961496765(0) win 32767 <mss 16396,sackOK,timestamp 1145181 0,nop,wscale 2>
21:24:40.784337 IP 127.0.0.1.1708 > 127.0.0.1.1709: S 3956628611:3956628611(0) ack 3961496766 win 32767 <mss 16396,sackOK,timestamp 1145181 1145181,nop,wscale 2

Those connections (to port 3785 or port 1708) are clearly to ports outside of the service port range:

bash-3.00# nsrports
Service ports: 7937-9936 
Connection ports: 0-0

This is a change from the old Legato NetWorker behavior, since in the past those ports, used during scheduled backups, were in the service port range.

2


David Stes
2006-01-01