XID Filtering with IPFilter

David Stes
email: stes@pandora.be

January 1, 2008

Abstract:

This paper describes an extension to IPFilter, to support the SUN RPC protocol. IPFilter is a firewall software, written by Darren Reed, and ported to various platforms, such as Solaris, HPUX, BSD and Linux. We show how we add some keywords to the IPFilter syntax to support filtering SUN RPC calls. The concept of IPFilter ``state'' is extended to keep track of SUN RPC program number and SUN RPC ``transaction ID'' (XID) numbers. When the firewall detects an RPC call, as described in the RFC 1831 internet standard, it waits for, and accepts, the corresponding response (with the same XID). Both UDP and TCP protocols have been implemented in a test implementation. As an example of SUN RPC over TCP, we discuss manual, client initiated, backups of EMC NetWorker.

Installing the software

The software described here, was developed on Linux Slackware 12.0.0 (using the kernel 2.6.21.5-smp kernel).

root@gecko:/opt/ip_fil4.1.27-stes# uname -a
Linux gecko 2.6.21.5-smp #2 SMP Tue Jun 19 14:58:11 CDT 2007
i686 Genuine Intel(R) CPU 2160  @ 1.80GHz GenuineIntel GNU/Linux

We started out with IPFilter version 4.1.27 and copied the source code to a directory ip_fil4.1.27-stes containing all of our modifications.

These modifications are available as patch (and are sent to Darren Reed, author of ipfilter).

The Linux Slackware port was relatively straightforward. One issue to consider is that IPFilter uses the libelf library, which is not included on Slackware by default. We installed a copy of libelf that was written by Michael Riepe :

ftp://ftp.ibiblio.org/pub/Linux/libs/

IPFilter uses some special files. Because the recent Linux kernel use udevd (and no devfs any longer), the simplest thing to do is to modify the IPFilter script to create the special files in the directory,

/lib/udev/devices

This is so that they are persistent across reboots.

An interesting option to set when compiling IPFilter is the support for CURSES. This supports the ipfstat option to display the open connections (states) in a manner similar to top :

# ipf -Fs
# save -s asterix /etc/motd
/etc/motd
/etc/
/

save: /etc/motd  6 KB 00:00:00      3 files
# ipfstat -tn
               gecko - IP Filter: v4.1.27-rpc - state top               00:00:56

Src: 0.0.0.0, Dest: 0.0.0.0, Proto: any, Sorted by: # bytes

Source IP             Destination IP         ST   PR   #pkts    #bytes       ttl
172.16.0.10,22512     172.16.0.9,8886       A/7  tcp      21     14176      3:58
172.16.0.10,28139     172.16.0.9,8442       A/7  tcp      11      8248      3:58
172.16.0.10,16344     172.16.0.9,8945       A/7  tcp      15      7632      3:58
127.0.0.1,49208       127.0.0.1,7937        A/7  tcp      18      2312      3:58
172.16.0.10,14956     172.16.0.9,8886       A/7  tcp      10      1976      3:58
127.0.0.1,25310       127.0.0.1,7937        A/7  tcp      12      1336      3:58
127.0.0.1,22747       127.0.0.1,7937        A/7  tcp      12      1328      3:58
127.0.0.1,28011       127.0.0.1,7937        A/7  tcp      12      1328      3:58
127.0.0.1,52703       127.0.0.1,7938        A/7  tcp      12       760      3:58
172.16.0.10,20102     172.16.0.9,7938       A/7  tcp       6       380      3:58
172.16.0.10,24521     172.16.0.9,7938       A/7  tcp       6       380      3:58
172.16.0.10,24682     172.16.0.9,7938       A/7  tcp       6       380      3:58
172.16.0.10,13995     172.16.0.9,7938       A/7  tcp       6       380      3:58

The above state table shows IPFilter states during (or just after) an EMC NetWorker save session.

Configuring the RPC firewall

As described in the internet standard RFC 1831, the RPC port numbers in the public range are managed by rpc@sun.com and should be identical for all sites.

Symbolic names for these port numbers are in the file :

/etc/rpc

The first extension to IPFilter is an extension to ippool so that it is possible to define ranges of RPC program numbers :

With the following file, /etc/ippool.nsr :

table role = ipf type = rpcs name = udppool
 { portmapper;nsrd;nsrstat;nsrjb;390120; };

table role = ipf type = rpcs name = tcppool
 { portmapper;nsrd;nsrmmd;nsrindexd;nsrmmdbd;nsrstat;nsrjb;390120; };

The following command defines tcppool and udppool :

ippool -f /etc/ippool.nsr

This is just a symbolic definition, so that tcppool and udppool are symbolic names that refer to some EMC NetWorker RPC program numbers and the portmapper, RPC 100000.

Next, we define an IPFilter configuration file, /etc/ipf.conf, so that only matching RPC remote calls are accepted using XID tracking :

pass in quick proto tcp from any to any port = 22
pass in quick proto udp all keep state (rpc in udppool)
pass in quick proto tcp from any to any keep state (rpc in tcppool)
block return-rst in on eth0 all

The above configfile says that SSH traffic is accepted without stateful packet filtering. No state information will be kept for SSH connections, these are simply accepted.

All other UDP and TCP connections will create a state entry in IPFilter. Because the RPC option is set on these states (the ``rpc'' keyword is an option between parentheses), IPFilter will check the RPC program number and remember the XID of a remote call. When the response arrives (with the same XID) the packets are matched to a state entry, and are accepted.

# ipf -Fi

# ipf -f /etc/ipf.conf

# ipf -T list
ipf_rpc_debug   min 0   max 0xa current 2
ipf_rpc_aware   min 0   max 0x1 current 1

# ipf -T ipf_rpc_debug=1

root@gecko:/opt# ipfstat -in
@1 pass in quick proto tcp from any to any port = ssh
@2 pass in quick proto udp from any to any keep state ( rpc in udppool )
@3 pass in quick proto tcp from any to any keep state ( rpc in tcppool )
@4 block return-rst in on eth0 all

Note that the debugging options will be used in the following sections.

RPC over UDP

In this section, we discuss stateful RPC filtering on UDP packets.

As can be seen in the following table, some RPC programs use UDP :

root@gecko:/opt# rpcinfo -p localhost
   program vers proto   port
    100000    2   tcp    111  portmapper
    100024    1   udp  32768  status
    100024    1   tcp  35422  status
    390113    1   tcp   7937  nsrexec
    390103    2   tcp   8727  nsrd
    390109    2   tcp   8727  nsrstat
    390110    1   tcp   8727  nsrjb
    390120    1   tcp   8727
    390103    2   udp   9779  nsrd
    390109    2   udp   9779  nsrstat
    390110    1   udp   9779  nsrjb
    390120    1   udp   9779
    390107    5   tcp   8612  nsrmmdbd
    390107    6   tcp   8612  nsrmmdbd
    390105    5   tcp   8960  nsrindexd
    390105    6   tcp   8960  nsrindexd

On a different client, we run ``rpcinfo -u gecko nsrstat'' to make an RPC call to procedure 0 on RPC nsrstat on the specified host (gecko) using UDP. In other words, with the -u option we force a transport of UDP.

# rpcinfo -u gecko nsrstat
program 390109 version 2 ready and waiting

Using ``tcpdump'', we can see that the following packets are sent between the hosts :

19:38:07.545885 IP 172.16.0.9.963 > 172.16.0.10.111: UDP, length 56
19:38:07.546024 IP 172.16.0.10.111 > 172.16.0.9.963: UDP, length 28
19:38:07.546385 IP 172.16.0.9.32881 > 172.16.0.10.9779: UDP, length 40
19:38:07.546426 IP 172.16.0.10.9779 > 172.16.0.9.32881: UDP, length 32
19:38:07.546885 IP 172.16.0.9.964 > 172.16.0.10.111: UDP, length 56
19:38:07.546972 IP 172.16.0.10.111 > 172.16.0.9.964: UDP, length 28
19:38:07.547384 IP 172.16.0.9.32881 > 172.16.0.10.9779: UDP, length 40
19:38:07.547410 IP 172.16.0.10.9779 > 172.16.0.9.32881: UDP, length 24

In debug mode, the IPFilter packet filter logs the following :

root@gecko:/opt/ip_fil4.1.27-stes# ipf -T ipf_rpc_debug=1
root@gecko:/opt/ip_fil4.1.27-stes# dmesg
IP Filter: v4.1.27-rpc unloaded
IP Filter: v4.1.27-rpc initialized.  Default = pass all, Logging = enabled
udp addstate   f79d5600 rpc 100000 xid 1308241280 state wait-response
udp checkstate f79d5600 rpc 100000 xid 1308241280 state accept
udp addstate   f79d5400 rpc 390109 xid 1814940010 state wait-response
udp checkstate f79d5400 rpc 390109 xid 1814940010 state accept
udp addstate   f79d5c00 rpc 100000 xid 1939701612 state wait-response
udp checkstate f79d5c00 rpc 100000 xid 1939701612 state accept
udp checkstate f79d5400 rpc 390109 xid 1497425995 state wait-response
udp checkstate f79d5400 rpc 390109 xid 1497425995 state accept

As can be seen from the above about, 3 different state entries are created. For the first state, program number 100000 is detected with a certain XID, and the corresponding reply (from the portmapper) is accepted.

Next, a state entry f79d5400 is created for RPC program number 390109 (nsrstat) and this is accepted by the firewall and matched against a response.

A new call to the portmapper is made, and then the existing state f79d5400 is reused because a new RPC call to ``nsrstat'' is made with the same UDP endpoints :

172.16.0.10.9779 
172.16.0.9.32881

In any case, as can be seen from the tcpdump output, the UDP case is fairly simple and straightforward.

RPC over TCP

The TCP case is of practical interest due to EMC NetWorker (and other applications) using RPC over TCP.

On a different client, we run ``rpcinfo -t gecko nsrstat'' to make an RPC call to procedure 0 on RPC nsrstat on the specified host (gecko) using TCP. In other words, with the -t option we force a transport of TCP.

bash-3.1# rpcinfo -t gecko nsrstat
program 390109 version 2 ready and waiting

The entire tcpdump is very large (even for this simple example), but we can study the initial part of it :

11:17:16.916958 IP 172.16.0.9.38669 > 172.16.0.10.111: S 2893649493:2893649493(0
) win 5840 <mss 1460,sackOK,timestamp 869016 0,nop,wscale 6>
11:17:16.917000 IP 172.16.0.10.111 > 172.16.0.9.38669: S 3494021617:3494021617(0
) ack 2893649494 win 5792 <mss 1460,sackOK,timestamp 966667512 869016,nop,wscale
 6>
11:17:16.917457 IP 172.16.0.9.38669 > 172.16.0.10.111: . ack 1 win 92 <nop,nop,t
imestamp 869016 966667512>
11:17:16.917482 IP 172.16.0.9.38669 > 172.16.0.10.111: P 1:61(60) ack 1 win 92 <
nop,nop,timestamp 869016 966667512>
11:17:16.917499 IP 172.16.0.10.111 > 172.16.0.9.38669: . ack 61 win 91 <nop,nop,
timestamp 966667512 869016>
11:17:16.917691 IP 172.16.0.10.111 > 172.16.0.9.38669: P 1:33(32) ack 61 win 91
<nop,nop,timestamp 966667512 869016>

It can be seen that there are first SYN, SYN/ACK and ACK packets being sent. The advertised MSS is 1460 bytes.

The RPC header for the remote procedure call nicely fits, in this particular, but fairly common case, into the fourth packet (TCP flag PUSH set). The response follows in the reverse direction.

We modified the IPFilter state code to capture RPC call and XID from these packets and to transition the state from ``wait for RPC call'' to ``wait for RPC response'' to finally, ``accepted'' (or ``rejected'').

For the ``rpcinfo -t gecko nsrstat'' command, we obtain four IPFilter state entries :

tcp addstate   c1919c00 state wait-rpc
tcp checkstate c1919c00 rpc 100000 xid 240491959 rev 1 state accept
tcp addstate   f7a25e00 state wait-rpc
tcp checkstate f7a25e00 rpc 390109 xid 1990326997 rev 1 state accept
tcp addstate   f7a25600 state wait-rpc
tcp checkstate f7a25600 rpc 100000 xid 2082515963 rev 1 state accept
tcp addstate   f7a25c00 state wait-rpc
tcp checkstate f7a25c00 rpc 390109 xid 1089952558 rev 1 state accept

The above is only a short view of the TCP session, as there are more states than just ``wait-rpc'' and ``accept'' in the TCP case.

If we increase the debug level, we can see, for the first state that there are many IP packets matching the state :

root@gecko:/home/stes# ipf -T ipf_rpc_debug=2

tcp addstate   c1919a00 state wait-rpc
tcp checkstate c1919a00 rpc 0 xid 0 rev 1 state wait-rpc
tcp checkstate c1919a00 rpc 0 xid 0 rev 0 state wait-rpc
tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 0 state wait-response
tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state wait-response
tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state accept
tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state accept
tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 0 state accept
tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 0 state accept
tcp addstate   f79dba00 state wait-rpc
tcp checkstate f79dba00 rpc 0 xid 0 rev 1 state wait-rpc
tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state accept
tcp checkstate f79dba00 rpc 0 xid 0 rev 0 state wait-rpc

When the SYN packet is sent, the state goes into ``wait-rpc''. Once the XID and RPC number is seen in the first real data packet, the state moves to ``wait-response''. It is only when the XID of the response is matched, that the state goes into ``accept'' mode.

Limiting port ranges for EMC NetWorker

In the case of EMC NetWorker, the command ``nsrports'' returns a range of ports that is being used by the EMC NetWorker RPC portmapper :

# Service ports: 7937-9936
# Connection ports: 10001-30000

Therefore, we can rewrite our IPFilter rule as follows :

pass in quick proto tcp from any to any port = 22
pass in quick proto udp all keep state (rpc in udppool)
pass in quick proto tcp from 172.16.0/24 to asterix port 7936:9937 keep state (rpc in tcppool)
block return-rst in on eth0 all

This merely limits the range on which we apply the RPC XID tracking, to a specific host (backup server, storage node etc.) or a specific DMZ subnet.

If one removes the option, by removing the (rpc) keyword between the parentheses, one obtains the EMC recommended way to configure firewalls (by opening the nsrports range).

The XID tracking simply allows an additional layer of security by controlling what type of traffic is going over this range (it should be RPC with program numbers equal to nsrd, nsrmmd, nsrindexd etc.).

Client initiated backup

A practical example of the IPFilter firewall, is using EMC NetWorker to do a client initiated (manual) backup.

Scheduled backups use a different protocol than RPC (this was discussed in previous papers). The scheduled backup protocol is, for EMC NetWorker 7.2, essentially BSD REXEC; this protocol can be firewalled with IPFilter but we limit ourselves here to client initiated backups and a discussion of RPC.

If we save a file on client ``gecko'' on backup server ``asterix'',

# save -s asterix /etc/passwd
/etc/passwd
/etc/
/

save: /etc/passwd  6 KB 00:00:00      3 files

First we see a TCP portmapper call and a call to 390113, the agent nsrexecd :

tcp addstate   f7adc800 state wait-rpc
tcp checkstate f7adc800 rpc 100000 xid 1182286540 rev 1 state accept
tcp addstate   f79de800 state wait-rpc
tcp checkstate f79de800 rpc 390113 xid 1182293408 rev 1 state accept

Next, we see a TCP portmapper call followed by a call to nsrstat :

tcp addstate   f79de600 state wait-rpc
tcp checkstate f79de600 rpc 100000 xid 1182291276 rev 1 state accept
tcp addstate   f7b6be00 state wait-rpc
tcp checkstate f7b6be00 rpc 390109 xid 1182290245 rev 1 state accept

The IPFilter firewall nicely tracks the XID of these calls and responses.

The actual save involves then RPC calls and responses to 390103 (nsrd, the backup server), 390113 (nsrexecd) and of course 390105 (nsrindexd to update the index) and 390104 (nsrmmd, the media multiplexing daemon which manages a particular tape drive) :

tcp addstate   f7b6ba00 state wait-rpc
tcp checkstate f7b6ba00 rpc 100000 xid 1182272343 rev 1 state accept
tcp addstate   f7b6bc00 state wait-rpc
tcp checkstate f7b6bc00 rpc 390103 xid 1182271027 rev 1 state accept
tcp addstate   f7b6b000 state wait-rpc
tcp checkstate f7b6b000 rpc 390113 xid 1182274254 rev 1 state accept
tcp addstate   f7b6b400 state wait-rpc
tcp checkstate f7b6b400 rpc 390113 xid 1182397966 rev 1 state accept
tcp addstate   f7b6b200 state wait-rpc
tcp checkstate f7b6b200 rpc 100000 xid 1182390616 rev 1 state accept
tcp addstate   f7bdca00 state wait-rpc
tcp addstate   f7bdc600 state wait-rpc
tcp checkstate f7bdc600 rpc 390113 xid 1182389058 rev 1 state accept
tcp addstate   f7bdcc00 state wait-rpc
tcp checkstate f7bdcc00 rpc 100000 xid 1182382717 rev 1 state accept
tcp addstate   f7bdce00 state wait-rpc
tcp checkstate f7bdce00 rpc 390105 xid 1182381681 rev 1 state accept
tcp checkstate f7bdca00 rpc 390104 xid 1182389551 rev 1 state accept

EMC NetWorker restore

The following example discusses the IPFilter states of EMC NetWorker recover (restore) over a firewall.

recover -s asterix

tcp addstate   f740b000 state wait-rpc
tcp checkstate f740b000 rpc 100000 xid 1181931620 rev 1 state accept
tcp addstate   f7a2be00 state wait-rpc
tcp checkstate f7a2be00 rpc 390113 xid 1181931685 rev 1 state accept
tcp addstate   f79d9800 state wait-rpc
tcp checkstate f79d9800 rpc 100000 xid 1181929966 rev 1 state accept
tcp addstate   f79d9e00 state wait-rpc
tcp checkstate f79d9e00 rpc 390109 xid 1181920761 rev 1 state accept
tcp addstate   f7a2ba00 state wait-rpc
tcp checkstate f7a2ba00 rpc 100000 xid 1181921707 rev 1 state accept
tcp addstate   f7a2b000 state wait-rpc
tcp checkstate f7a2b000 rpc 390103 xid 1181912205 rev 1 state accept
tcp addstate   f7a2b800 state wait-rpc
tcp checkstate f7a2b800 rpc 390113 xid 1181915588 rev 1 state accept
tcp addstate   f7a2b600 state wait-rpc
tcp checkstate f7a2b600 rpc 100000 xid 1181902472 rev 1 state accept
tcp addstate   f7a2b400 state wait-rpc
tcp checkstate f7a2b400 rpc 390105 xid 1181901461 rev 1 state accept
tcp addstate   c1919400 state wait-rpc
tcp checkstate c1919400 rpc 390113 xid 1181897917 rev 1 state accept

When launching the restore, we see RPC calls to 10000, portmapper, 390109, nsrstat, 390105 (the index daemon to find the file in the case of browsable restore), and 390113, the nsrexecd agent.

Finally the restore reports :

/opt/ not in index
<return> will exit.
recover: Current working directory is /etc/
recover> 
recover> add /etc/passwd
/etc
1 file(s) marked for recovery
recover>

At this point, no new IPFilter state is created, and there are simply packets flowing over the established TCP connections.

recover> relocate /tmp
recover> 

tcp addstate   f79d5600 state wait-rpc
tcp checkstate f79d5600 rpc 100000 xid 1181895609 rev 1 state accept
tcp addstate   f79d5c00 state wait-rpc
tcp checkstate f79d5c00 rpc 390109 xid 1181894586 rev 1 state accept

This triggers again some RPC TCP calls to the (local) agent which in our example, is running on the firewall itself. Finally we launch the restore :

recover> recover
recover: Total estimated disk space needed for recover is 4 KB.
Recovering 1 file from /etc/ into /tmp
Volumes needed (all on-line):
        DISK1 at /disk1
Requesting 1 file(s), this may take a while...
./passwd
./passwd file exists, overwrite (n, y, N, Y) or rename (r, R) [n]? y
overwriting ./passwd
Received 1 file(s) from NSR server `asterix'
Recover completion time: Tue Jan  1 22:12:52 2008
recover> quit

tcp addstate   f7adba00 state wait-rpc
tcp checkstate f7adba00 rpc 390113 xid 1182760090 rev 1 state accept
tcp addstate   f7adb800 state wait-rpc
tcp checkstate f7adb800 rpc 100000 xid 1182759348 rev 1 state accept
tcp addstate   f7adbc00 state wait-rpc
tcp checkstate f7adbc00 rpc 390103 xid 1182758313 rev 1 state accept
tcp addstate   f7a2da00 state wait-rpc
tcp checkstate f7a2da00 rpc 100000 xid 1182745799 rev 1 state accept
tcp addstate   f7a2dc00 state wait-rpc
tcp checkstate f7a2dc00 rpc 390104 xid 1182744608 rev 1 state accept

It can be seen that only at this point, RPC calls are being made to nsrd, 390103 and the media daemon nsrmmd, 390104 to restore the file from tape.



David Stes 2008-01-02