David Stes
Molenstraat 5
2018 Antwerp, Flanders, Belgium
email: stes@pandora.be
January 1, 2008
The software described here, was developed on Linux Slackware 12.0.0 (using the kernel 2.6.21.5-smp kernel).
root@gecko:/opt/ip_fil4.1.27-stes# uname -a Linux gecko 2.6.21.5-smp #2 SMP Tue Jun 19 14:58:11 CDT 2007 i686 Genuine Intel(R) CPU 2160 @ 1.80GHz GenuineIntel GNU/Linux
We started out with IPFilter version 4.1.27 and copied the source code to a directory ip_fil4.1.27-stes containing all of our modifications.
These modifications are available as patch (and are sent to Darren Reed, author of ipfilter).
The Linux Slackware port was relatively straightforward. One issue to consider is that IPFilter uses the libelf library, which is not included on Slackware by default. We installed a copy of libelf that was written by Michael Riepe :
ftp://ftp.ibiblio.org/pub/Linux/libs/
IPFilter uses some special files. Because the recent Linux kernel use udevd (and no devfs any longer), the simplest thing to do is to modify the IPFilter script to create the special files in the directory,
/lib/udev/devices
This is so that they are persistent across reboots.
An interesting option to set when compiling IPFilter is the support for CURSES. This supports the ipfstat option to display the open connections (states) in a manner similar to top :
# ipf -Fs
# save -s asterix /etc/motd
/etc/motd
/etc/
/
save: /etc/motd 6 KB 00:00:00 3 files
# ipfstat -tn
gecko - IP Filter: v4.1.27-rpc - state top 00:00:56
Src: 0.0.0.0, Dest: 0.0.0.0, Proto: any, Sorted by: # bytes
Source IP Destination IP ST PR #pkts #bytes ttl
172.16.0.10,22512 172.16.0.9,8886 A/7 tcp 21 14176 3:58
172.16.0.10,28139 172.16.0.9,8442 A/7 tcp 11 8248 3:58
172.16.0.10,16344 172.16.0.9,8945 A/7 tcp 15 7632 3:58
127.0.0.1,49208 127.0.0.1,7937 A/7 tcp 18 2312 3:58
172.16.0.10,14956 172.16.0.9,8886 A/7 tcp 10 1976 3:58
127.0.0.1,25310 127.0.0.1,7937 A/7 tcp 12 1336 3:58
127.0.0.1,22747 127.0.0.1,7937 A/7 tcp 12 1328 3:58
127.0.0.1,28011 127.0.0.1,7937 A/7 tcp 12 1328 3:58
127.0.0.1,52703 127.0.0.1,7938 A/7 tcp 12 760 3:58
172.16.0.10,20102 172.16.0.9,7938 A/7 tcp 6 380 3:58
172.16.0.10,24521 172.16.0.9,7938 A/7 tcp 6 380 3:58
172.16.0.10,24682 172.16.0.9,7938 A/7 tcp 6 380 3:58
172.16.0.10,13995 172.16.0.9,7938 A/7 tcp 6 380 3:58
The above state table shows IPFilter states during (or just after) an EMC NetWorker save session.
As described in the internet standard RFC 1831, the RPC port numbers in the public range are managed by rpc@sun.com and should be identical for all sites.
Symbolic names for these port numbers are in the file :
/etc/rpc
The first extension to IPFilter is an extension to ippool so that it is possible to define ranges of RPC program numbers :
With the following file, /etc/ippool.nsr :
table role = ipf type = rpcs name = udppool
{ portmapper;nsrd;nsrstat;nsrjb;390120; };
table role = ipf type = rpcs name = tcppool
{ portmapper;nsrd;nsrmmd;nsrindexd;nsrmmdbd;nsrstat;nsrjb;390120; };
The following command defines tcppool and udppool :
ippool -f /etc/ippool.nsr
This is just a symbolic definition, so that tcppool and udppool are symbolic names that refer to some EMC NetWorker RPC program numbers and the portmapper, RPC 100000.
Next, we define an IPFilter configuration file, /etc/ipf.conf, so that only matching RPC remote calls are accepted using XID tracking :
pass in quick proto tcp from any to any port = 22 pass in quick proto udp all keep state (rpc in udppool) pass in quick proto tcp from any to any keep state (rpc in tcppool) block return-rst in on eth0 all
The above configfile says that SSH traffic is accepted without stateful packet filtering. No state information will be kept for SSH connections, these are simply accepted.
All other UDP and TCP connections will create a state entry in IPFilter. Because the RPC option is set on these states (the ``rpc'' keyword is an option between parentheses), IPFilter will check the RPC program number and remember the XID of a remote call. When the response arrives (with the same XID) the packets are matched to a state entry, and are accepted.
# ipf -Fi # ipf -f /etc/ipf.conf # ipf -T list ipf_rpc_debug min 0 max 0xa current 2 ipf_rpc_aware min 0 max 0x1 current 1 # ipf -T ipf_rpc_debug=1 root@gecko:/opt# ipfstat -in @1 pass in quick proto tcp from any to any port = ssh @2 pass in quick proto udp from any to any keep state ( rpc in udppool ) @3 pass in quick proto tcp from any to any keep state ( rpc in tcppool ) @4 block return-rst in on eth0 all
Note that the debugging options will be used in the following sections.
In this section, we discuss stateful RPC filtering on UDP packets.
As can be seen in the following table, some RPC programs use UDP :
root@gecko:/opt# rpcinfo -p localhost
program vers proto port
100000 2 tcp 111 portmapper
100024 1 udp 32768 status
100024 1 tcp 35422 status
390113 1 tcp 7937 nsrexec
390103 2 tcp 8727 nsrd
390109 2 tcp 8727 nsrstat
390110 1 tcp 8727 nsrjb
390120 1 tcp 8727
390103 2 udp 9779 nsrd
390109 2 udp 9779 nsrstat
390110 1 udp 9779 nsrjb
390120 1 udp 9779
390107 5 tcp 8612 nsrmmdbd
390107 6 tcp 8612 nsrmmdbd
390105 5 tcp 8960 nsrindexd
390105 6 tcp 8960 nsrindexd
On a different client, we run ``rpcinfo -u gecko nsrstat'' to make an RPC call to procedure 0 on RPC nsrstat on the specified host (gecko) using UDP. In other words, with the -u option we force a transport of UDP.
# rpcinfo -u gecko nsrstat program 390109 version 2 ready and waiting
Using ``tcpdump'', we can see that the following packets are sent between the hosts :
19:38:07.545885 IP 172.16.0.9.963 > 172.16.0.10.111: UDP, length 56 19:38:07.546024 IP 172.16.0.10.111 > 172.16.0.9.963: UDP, length 28 19:38:07.546385 IP 172.16.0.9.32881 > 172.16.0.10.9779: UDP, length 40 19:38:07.546426 IP 172.16.0.10.9779 > 172.16.0.9.32881: UDP, length 32 19:38:07.546885 IP 172.16.0.9.964 > 172.16.0.10.111: UDP, length 56 19:38:07.546972 IP 172.16.0.10.111 > 172.16.0.9.964: UDP, length 28 19:38:07.547384 IP 172.16.0.9.32881 > 172.16.0.10.9779: UDP, length 40 19:38:07.547410 IP 172.16.0.10.9779 > 172.16.0.9.32881: UDP, length 24
In debug mode, the IPFilter packet filter logs the following :
root@gecko:/opt/ip_fil4.1.27-stes# ipf -T ipf_rpc_debug=1 root@gecko:/opt/ip_fil4.1.27-stes# dmesg IP Filter: v4.1.27-rpc unloaded IP Filter: v4.1.27-rpc initialized. Default = pass all, Logging = enabled udp addstate f79d5600 rpc 100000 xid 1308241280 state wait-response udp checkstate f79d5600 rpc 100000 xid 1308241280 state accept udp addstate f79d5400 rpc 390109 xid 1814940010 state wait-response udp checkstate f79d5400 rpc 390109 xid 1814940010 state accept udp addstate f79d5c00 rpc 100000 xid 1939701612 state wait-response udp checkstate f79d5c00 rpc 100000 xid 1939701612 state accept udp checkstate f79d5400 rpc 390109 xid 1497425995 state wait-response udp checkstate f79d5400 rpc 390109 xid 1497425995 state accept
As can be seen from the above about, 3 different state entries are created. For the first state, program number 100000 is detected with a certain XID, and the corresponding reply (from the portmapper) is accepted.
Next, a state entry f79d5400 is created for RPC program number 390109 (nsrstat) and this is accepted by the firewall and matched against a response.
A new call to the portmapper is made, and then the existing state f79d5400 is reused because a new RPC call to ``nsrstat'' is made with the same UDP endpoints :
172.16.0.10.9779 172.16.0.9.32881
In any case, as can be seen from the tcpdump output, the UDP case is fairly simple and straightforward.
The TCP case is of practical interest due to EMC NetWorker (and other applications) using RPC over TCP.
On a different client, we run ``rpcinfo -t gecko nsrstat'' to make an RPC call to procedure 0 on RPC nsrstat on the specified host (gecko) using TCP. In other words, with the -t option we force a transport of TCP.
bash-3.1# rpcinfo -t gecko nsrstat program 390109 version 2 ready and waiting
The entire tcpdump is very large (even for this simple example), but we can study the initial part of it :
11:17:16.916958 IP 172.16.0.9.38669 > 172.16.0.10.111: S 2893649493:2893649493(0 ) win 5840 <mss 1460,sackOK,timestamp 869016 0,nop,wscale 6> 11:17:16.917000 IP 172.16.0.10.111 > 172.16.0.9.38669: S 3494021617:3494021617(0 ) ack 2893649494 win 5792 <mss 1460,sackOK,timestamp 966667512 869016,nop,wscale 6> 11:17:16.917457 IP 172.16.0.9.38669 > 172.16.0.10.111: . ack 1 win 92 <nop,nop,t imestamp 869016 966667512> 11:17:16.917482 IP 172.16.0.9.38669 > 172.16.0.10.111: P 1:61(60) ack 1 win 92 < nop,nop,timestamp 869016 966667512> 11:17:16.917499 IP 172.16.0.10.111 > 172.16.0.9.38669: . ack 61 win 91 <nop,nop, timestamp 966667512 869016> 11:17:16.917691 IP 172.16.0.10.111 > 172.16.0.9.38669: P 1:33(32) ack 61 win 91 <nop,nop,timestamp 966667512 869016>
It can be seen that there are first SYN, SYN/ACK and ACK packets being sent. The advertised MSS is 1460 bytes.
The RPC header for the remote procedure call nicely fits, in this particular, but fairly common case, into the fourth packet (TCP flag PUSH set). The response follows in the reverse direction.
We modified the IPFilter state code to capture RPC call and XID from these packets and to transition the state from ``wait for RPC call'' to ``wait for RPC response'' to finally, ``accepted'' (or ``rejected'').
For the ``rpcinfo -t gecko nsrstat'' command, we obtain four IPFilter state entries :
tcp addstate c1919c00 state wait-rpc tcp checkstate c1919c00 rpc 100000 xid 240491959 rev 1 state accept tcp addstate f7a25e00 state wait-rpc tcp checkstate f7a25e00 rpc 390109 xid 1990326997 rev 1 state accept tcp addstate f7a25600 state wait-rpc tcp checkstate f7a25600 rpc 100000 xid 2082515963 rev 1 state accept tcp addstate f7a25c00 state wait-rpc tcp checkstate f7a25c00 rpc 390109 xid 1089952558 rev 1 state accept
The above is only a short view of the TCP session, as there are more states than just ``wait-rpc'' and ``accept'' in the TCP case.
If we increase the debug level, we can see, for the first state that there are many IP packets matching the state :
root@gecko:/home/stes# ipf -T ipf_rpc_debug=2 tcp addstate c1919a00 state wait-rpc tcp checkstate c1919a00 rpc 0 xid 0 rev 1 state wait-rpc tcp checkstate c1919a00 rpc 0 xid 0 rev 0 state wait-rpc tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 0 state wait-response tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state wait-response tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state accept tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state accept tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 0 state accept tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 0 state accept tcp addstate f79dba00 state wait-rpc tcp checkstate f79dba00 rpc 0 xid 0 rev 1 state wait-rpc tcp checkstate c1919a00 rpc 100000 xid 1487799080 rev 1 state accept tcp checkstate f79dba00 rpc 0 xid 0 rev 0 state wait-rpc
When the SYN packet is sent, the state goes into ``wait-rpc''. Once the XID and RPC number is seen in the first real data packet, the state moves to ``wait-response''. It is only when the XID of the response is matched, that the state goes into ``accept'' mode.
In the case of EMC NetWorker, the command ``nsrports'' returns a range of ports that is being used by the EMC NetWorker RPC portmapper :
# Service ports: 7937-9936 # Connection ports: 10001-30000
Therefore, we can rewrite our IPFilter rule as follows :
pass in quick proto tcp from any to any port = 22 pass in quick proto udp all keep state (rpc in udppool) pass in quick proto tcp from 172.16.0/24 to asterix port 7936:9937 keep state (rpc in tcppool) block return-rst in on eth0 all
This merely limits the range on which we apply the RPC XID tracking, to a specific host (backup server, storage node etc.) or a specific DMZ subnet.
If one removes the option, by removing the (rpc) keyword between the parentheses, one obtains the EMC recommended way to configure firewalls (by opening the nsrports range).
The XID tracking simply allows an additional layer of security by controlling what type of traffic is going over this range (it should be RPC with program numbers equal to nsrd, nsrmmd, nsrindexd etc.).
A practical example of the IPFilter firewall, is using EMC NetWorker to do a client initiated (manual) backup.
Scheduled backups use a different protocol than RPC (this was discussed in previous papers). The scheduled backup protocol is, for EMC NetWorker 7.2, essentially BSD REXEC; this protocol can be firewalled with IPFilter but we limit ourselves here to client initiated backups and a discussion of RPC.
If we save a file on client ``gecko'' on backup server ``asterix'',
# save -s asterix /etc/passwd /etc/passwd /etc/ / save: /etc/passwd 6 KB 00:00:00 3 files
First we see a TCP portmapper call and a call to 390113, the agent nsrexecd :
tcp addstate f7adc800 state wait-rpc tcp checkstate f7adc800 rpc 100000 xid 1182286540 rev 1 state accept tcp addstate f79de800 state wait-rpc tcp checkstate f79de800 rpc 390113 xid 1182293408 rev 1 state accept
Next, we see a TCP portmapper call followed by a call to nsrstat :
tcp addstate f79de600 state wait-rpc tcp checkstate f79de600 rpc 100000 xid 1182291276 rev 1 state accept tcp addstate f7b6be00 state wait-rpc tcp checkstate f7b6be00 rpc 390109 xid 1182290245 rev 1 state accept
The IPFilter firewall nicely tracks the XID of these calls and responses.
The actual save involves then RPC calls and responses to 390103 (nsrd, the backup server), 390113 (nsrexecd) and of course 390105 (nsrindexd to update the index) and 390104 (nsrmmd, the media multiplexing daemon which manages a particular tape drive) :
tcp addstate f7b6ba00 state wait-rpc tcp checkstate f7b6ba00 rpc 100000 xid 1182272343 rev 1 state accept tcp addstate f7b6bc00 state wait-rpc tcp checkstate f7b6bc00 rpc 390103 xid 1182271027 rev 1 state accept tcp addstate f7b6b000 state wait-rpc tcp checkstate f7b6b000 rpc 390113 xid 1182274254 rev 1 state accept tcp addstate f7b6b400 state wait-rpc tcp checkstate f7b6b400 rpc 390113 xid 1182397966 rev 1 state accept tcp addstate f7b6b200 state wait-rpc tcp checkstate f7b6b200 rpc 100000 xid 1182390616 rev 1 state accept tcp addstate f7bdca00 state wait-rpc tcp addstate f7bdc600 state wait-rpc tcp checkstate f7bdc600 rpc 390113 xid 1182389058 rev 1 state accept tcp addstate f7bdcc00 state wait-rpc tcp checkstate f7bdcc00 rpc 100000 xid 1182382717 rev 1 state accept tcp addstate f7bdce00 state wait-rpc tcp checkstate f7bdce00 rpc 390105 xid 1182381681 rev 1 state accept tcp checkstate f7bdca00 rpc 390104 xid 1182389551 rev 1 state accept
The following example discusses the IPFilter states of EMC NetWorker recover (restore) over a firewall.
recover -s asterix tcp addstate f740b000 state wait-rpc tcp checkstate f740b000 rpc 100000 xid 1181931620 rev 1 state accept tcp addstate f7a2be00 state wait-rpc tcp checkstate f7a2be00 rpc 390113 xid 1181931685 rev 1 state accept tcp addstate f79d9800 state wait-rpc tcp checkstate f79d9800 rpc 100000 xid 1181929966 rev 1 state accept tcp addstate f79d9e00 state wait-rpc tcp checkstate f79d9e00 rpc 390109 xid 1181920761 rev 1 state accept tcp addstate f7a2ba00 state wait-rpc tcp checkstate f7a2ba00 rpc 100000 xid 1181921707 rev 1 state accept tcp addstate f7a2b000 state wait-rpc tcp checkstate f7a2b000 rpc 390103 xid 1181912205 rev 1 state accept tcp addstate f7a2b800 state wait-rpc tcp checkstate f7a2b800 rpc 390113 xid 1181915588 rev 1 state accept tcp addstate f7a2b600 state wait-rpc tcp checkstate f7a2b600 rpc 100000 xid 1181902472 rev 1 state accept tcp addstate f7a2b400 state wait-rpc tcp checkstate f7a2b400 rpc 390105 xid 1181901461 rev 1 state accept tcp addstate c1919400 state wait-rpc tcp checkstate c1919400 rpc 390113 xid 1181897917 rev 1 state accept
When launching the restore, we see RPC calls to 10000, portmapper, 390109, nsrstat, 390105 (the index daemon to find the file in the case of browsable restore), and 390113, the nsrexecd agent.
Finally the restore reports :
/opt/ not in index <return> will exit. recover: Current working directory is /etc/ recover> recover> add /etc/passwd /etc 1 file(s) marked for recovery recover>
At this point, no new IPFilter state is created, and there are simply packets flowing over the established TCP connections.
recover> relocate /tmp recover> tcp addstate f79d5600 state wait-rpc tcp checkstate f79d5600 rpc 100000 xid 1181895609 rev 1 state accept tcp addstate f79d5c00 state wait-rpc tcp checkstate f79d5c00 rpc 390109 xid 1181894586 rev 1 state accept
This triggers again some RPC TCP calls to the (local) agent which in our example, is running on the firewall itself. Finally we launch the restore :
recover> recover
recover: Total estimated disk space needed for recover is 4 KB.
Recovering 1 file from /etc/ into /tmp
Volumes needed (all on-line):
DISK1 at /disk1
Requesting 1 file(s), this may take a while...
./passwd
./passwd file exists, overwrite (n, y, N, Y) or rename (r, R) [n]? y
overwriting ./passwd
Received 1 file(s) from NSR server `asterix'
Recover completion time: Tue Jan 1 22:12:52 2008
recover> quit
tcp addstate f7adba00 state wait-rpc
tcp checkstate f7adba00 rpc 390113 xid 1182760090 rev 1 state accept
tcp addstate f7adb800 state wait-rpc
tcp checkstate f7adb800 rpc 100000 xid 1182759348 rev 1 state accept
tcp addstate f7adbc00 state wait-rpc
tcp checkstate f7adbc00 rpc 390103 xid 1182758313 rev 1 state accept
tcp addstate f7a2da00 state wait-rpc
tcp checkstate f7a2da00 rpc 100000 xid 1182745799 rev 1 state accept
tcp addstate f7a2dc00 state wait-rpc
tcp checkstate f7a2dc00 rpc 390104 xid 1182744608 rev 1 state accept
It can be seen that only at this point, RPC calls are being made to nsrd, 390103 and the media daemon nsrmmd, 390104 to restore the file from tape.