David Stes
Molenstraat 5
2018 Antwerp, Flanders, Belgium
email: stes@pandora.be
September 24, 2008
The following ip addresses are used:
172.16.0.8 gecko 192.168.0.8 gecko-bge0 172.16.0.100 newt 192.168.0.5 darkstar
Our backup server is a Linux system called darkstar. It is running EMC NetWorker version 7.4 SP3.
# nsradmin -s darkstar -v1 -p nsrexec
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin>print
type: NSRLA;
name: darkstar;
version: EMC NetWorker 7.4.3.Build.569 08/28/08;
The client software of EMC NetWorker 7.4 SP3 registers the following SUN RPC ports, as can be shown using our nwrpcinfo tool :
bash-3.00# nwrpcinfo -p
program vers proto port
100000 2 tcp 7938 portmapper
100000 2 udp 7938 portmapper
390436 1 tcp 8194
390435 1 tcp 9024
390113 1 tcp 7937 nsrexecd
The 390435 and 390436 are unnamed, and not further discussed by EMC, but these programs were introduced in EMC NetWorker 7.3 (see our paper of 2006 on netfilter firewalling); they are related to authserver. On our backup server, we use the default auth methods, although that in practice it may be preferable to use oldauth only :
nsradmin> update auth methods: "0.0.0.0/0,nsrauth/oldauth"
auth methods: "0.0.0.0/0,nsrauth/oldauth";
Update? y
The nwrpcinfo tool is a modification of the standard rpcinfo. It is compiled from the Linux libc C sources but with a modification to query the EMC portmapper, by using PMAPPORT 7938 instead of 111 :
/usr/include/rpc/pmap_prot.h:#define PMAPPORT ((u_short)111) /usr/include/rpc/pmap_prot.h:#define PMAPPORT ((u_short)7938) /* textdomain (_libc_intl_domainname); don't use gettext */
The TCP and UDP port numbers that EMC NetWorker uses are dynamically allocated from a range of ports. For example, for each (enabled) tape drive, a different nsrmmd process is created with a different TCP address, and EMC NetWorker will listen on that port for data that needs to be written to that specific tape unit.
After labeling a disk device, so that we have a volume to do backups, the following SUN RPC program numbers are registered on our backup server :
bash-3.00# nwrpcinfo -p
program vers proto port
100000 2 tcp 7938 portmapper
100000 2 udp 7938 portmapper
390436 1 tcp 8194
390435 1 tcp 9024
390113 1 tcp 7937 nsrexecd
390103 2 tcp 9141 nsrd
390109 2 tcp 9141 nsrstat
390110 1 tcp 9141 nsrjbd
390120 1 tcp 9141
390109 2 udp 8865 nsrstat
390107 5 tcp 9389 nsrmmdbd
390107 6 tcp 9389 nsrmmdbd
390433 1 tcp 8824 nsrjobd
390105 5 tcp 8806 nsrindexd
390105 6 tcp 8806 nsrindexd
390104 105 tcp 8694 nsrmmd
In terms of service port range, we use the default range :
# nsrports -s darkstar Service ports: 7937-9936 Connection ports: 0-0
The SUN RPC program numbers for EMC NetWorker (formerly Legato) used to be managed by SUN, it is nowadays available at:
http://www.nfsv4-editor.org/rpc-numbers-1831bis.txt
The backup client is a different Linux system called newt. It is running EMC NetWorker version 6.1.3, which was at the time still a Legato product, so our client is Legato NetWorker 6.1.3.
Unfortunately, the EMC NetWorker 7.4 SP3 software prints out the following error when we try to make a command-line connection to the agent :
bash-3.00# nsradmin -s newt -p nsrexec -v1 39078:nsradmin: RPC error: Program not registered
The EMC NetWorker 7.4 SP3 software is complaining about the fact that there is no SUN RPC 390436 running on newt :
darkstar -> newt PORTMAP C GETPORT prog=390436 (?) vers=1 proto=TCP
darkstar -> newt PORTMAP C GETPORT prog=390436 (?) vers=1 proto=TCP
With the old nsradmin executable, we can still make a connection:
# nsradmin613 -s newt -p nsrexec -v1
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin613> print
type: NSRLA;
name: newt;
version: Legato NetWorker 6.1.3.Build.428 11/21/02;
The old executable is in some sense more powerful than the new one, since it also allows us to connect to the EMC NetWorker 7.4 software. This is due of course to the fact that the old executable doesn't issue a port request to the 390436 program. It could be argued that the behavior of the new nsradmin executable could be improved, so that is able to connect to older agents.
The client is running the following SUN RPC services:
# nwrpcinfo -p newt
program vers proto port
100000 2 tcp 7938 portmapper
100000 2 udp 7938 portmapper
390113 1 tcp 7937 nsrexecd
For the service port range, newt is also using the default settings:
# nsrports -s newt Service ports: 7937-9936 Connection ports: 10001-30000
The hostname of our router is gecko, it's a PCI-e system with two gigabit interfaces :
# dladm show-link LINK CLASS MTU STATE OVER e1000g0 phys 1500 up -- bge0 phys 1500 up --
One interface is in the same subnet as the EMC NetWorker backup server, and the other interface is in the subnet of the EMC NetWorker client :
stes@gecko:~# netstat -rn Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- 172.16.0.0 172.16.0.8 U 1 18 e1000g0 192.168.0.0 192.168.0.8 U 1 11 bge0 127.0.0.1 127.0.0.1 UH 1 0 lo0
The operating system is OpenSolaris 2008.05 :
stes@gecko:# uname -a SunOS gecko 5.11 snv_86 i86pc i386 i86pc Solaris
Because this system acts as a router, we enabled IP forwarding, and disabled the automatic configuration of network interfaces.
stes@gecko:# svcs -a | grep forward online 9:35:35 svc:/network/ipv4-forwarding:default stes@gecko:~# svcs -a | grep physical disabled 17:49:32 svc:/network/physical:nwam online 17:49:37 svc:/network/physical:default
By default, this system comes with version 4.1.9 of OpenSolaris IPFilter. However, we uninstalled the standard packages SUNWipfr and SUNWipfu (using the command pkg uninstall SUNWipf). Next, we compiled ourselves the latest version of OpenSolaris IPFilter, so the software that is described in this paper is version 5.0.4, which is a different branch of the software :
stes@gecko:~# pkginfo -l ipfx
PKGINST: ipfx
NAME: IP Filter (64-bit)
CATEGORY: system
ARCH: i386
VERSION: 5.0.4
BASEDIR: /
VENDOR: Darren Reed
DESC: This package contains tools for building a firewall
INSTDATE: Sep 21 2008 04:08
EMAIL: darrenr@pobox.com
The compilation itself works as follows on OpenSolaris :
svccfg export network/ipfilter > /tmp/ipfilter.def cp /lib/svc/method/ipfilter /tmp svcadm disable ipfilter pkgrm ipf ipfx make solaris cd SunOS5;make pkg make package svcadm enable ipfilter # ipf -E (if necessary, enable should do this)
The first approach to configure the OpenSolaris IPFilter is to allow TCP packets of the EMC NetWorker port range to pass, and we can write some rules for this. These rules use stateful inspection of the TCP sessions i.e., we request that OpenSolaris IPFilter stores the TCP state of each connection.
Check the firewall:
# ipf -V ipf: IP Filter: v5.0.4 (648) Kernel: IP Filter: v5.0.4 Running: no Log Flags: 0 = none set Default: pass all, Logging: available Active list: 0 Feature mask: 0x107
Because OpenSolaris IPFilter is not yet Running, we enable it and ask it to log blocked packets:
stes@gecko:~# ipf -E stes@gecko:~# ipf -l blocked stes@gecko:~# ipf -V ipf: IP Filter: v5.0.4 (648) Kernel: IP Filter: v5.0.4 Running: yes Log Flags: 0x20000000 = block Default: pass all, Logging: available Active list: 0 Feature mask: 0x107
Flush (empty) the list of firewall rules, then load the firewall rules, allowing SSH over the firewall and opening the ports in the range of 7937 to 9936.
stes@gecko:/etc/ipf# ipf -Fo stes@gecko:/etc/ipf# cat /etc/ipf/ipf.nsr pass out quick proto tcp from any to any port = ssh keep state pass out quick proto tcp from any to any port 7936:9937 keep state pass out quick proto udp from any to any port 7936:9937 keep state block out all stes@gecko:/etc/ipf# ipf -o -f /etc/ipf/ipf.nsr stes@gecko:/etc/ipf# ipfstat -on @1 pass out quick proto tcp from any to any port = ssh keep state @2 pass out quick proto tcp from any to any port 7936:9937 keep state @3 pass out quick proto udp from any to any port 7936:9937 keep state @4 block out all
After enabling this set of rules, it can be observed that ping doesn't work any longer between the two Linux systems, and that we can still run SSH over the firewall. The number of matches for rules can be displayed as follows:
stes@gecko:/etc/ipf# ipfstat -ho 1 pass out quick proto tcp from any to any port = ssh keep state 0 pass out quick proto tcp from any to any port 7936:9937 keep state 0 pass out quick proto udp from any to any port 7936:9937 keep state 2 block out all
As discussed in previous papers on EMC NetWorker, from a point of view of network protocols, EMC NetWorker uses two different protocols for manual and scheduled backups. We will first investigate here how these backups work with our firewall (that is configured as is traditionally done, by opening a range of ports).
To simplify things, it could be stated that basically, EMC NetWorker uses SUN RPC to transfer data from the client to the server during manual (client initiated) backups.
Again, simplifying things, it can be stated that essentially, EMC NetWorker uses a BSD rexec derived protocol, for the server to contact the client during a scheduled backup, where the server requests remote command execution of the command to run a manual backup.
This is a simplification, and many EMC NetWorker products heavily use SUN RPC for such things as initiating VSS or other PowerSnap snapshots etc. but the basic distinction between manual and scheduled backup is essential, in terms of network protocols.
First, we try the manual (client initiated) backup over our firewall:
root@newt:/# save -s darkstar /etc/motd /etc/motd /etc/ / save: /etc/motd 4 KB 00:00:01 3 files
The backup works and it can be observed that it uses various TCP sessions:
stes@gecko:/etc/ipf# ipfstat -ho 1 pass out quick proto tcp from any to any port = ssh keep state 16 pass out quick proto tcp from any to any port 7936:9937 keep state 0 pass out quick proto udp from any to any port 7936:9937 keep state 13 block out all
The case of scheduled backups is usually much more complicated with EMC NetWorker, as it involves a variety of protocols, depending on the client (and server) version.
Without firewall, it works fine:
bash-3.00# savegrp -v -c newt 32451:savegrp: newt:/etc/motd level=incr 7236:savegrp: Group will not limit job parallelism 32493:savegrp: newt:probe started savefs -s darkstar -c newt -g Default -p -l full -R -v -F /etc/motd savegrp:Default * newt:Probe See the file /nsr/tmp/sg/Default/sso.newt.6ZlZC0 for output of save command. 7340:savegrp: newt:probe succeeded. newt:/etc/motd level=full, dn=0, mx=1, vers=ssbrowse, p=12 32494:savegrp: newt:/etc/motd started save -s darkstar -g Default -LL -m newt -l full -W 78 -N /etc/motd /etc/motd savegrp:Default * newt:/etc/motd See the file /nsr/tmp/sg/Default/sso.newt.8KD4nj for output of save command.
Unfortunately, when we enable the firewall, the scheduled backup hangs. It can be seen that the OpenSolaris IPFilter is blocking the following packets:
# ipmon -a 24/09/2008 20:30:09.734429 e1000g0 @0:4 b 192.168.0.5,2687 -> 172.16.0.100,111 PR tcp len 20 60 -S OUT
The command ipmon shows some requests to the (SUN) portmapper 111, and this makes some sense, in the context of the EMC NetWorker protocols, so we add a rule specifically for this.
stes@gecko:/etc/ipf# cat /etc/ipf/ipf.nsr pass out quick proto tcp from any to any port = ssh keep state pass out quick proto tcp from any to any port = 111 keep state pass out quick proto udp from any to any port = 111 keep state pass out quick proto tcp from any to any port 7936:9937 keep state pass out quick proto udp from any to any port 7936:9937 keep state block out all stes@gecko:/etc/ipf# ipf -o -f /etc/ipf/ipf.nsr stes@gecko:/etc/ipf# ipfstat -on @1 pass out quick proto tcp from any to any port = ssh keep state @2 pass out quick proto tcp from any to any port = sunrpc keep state @3 pass out quick proto udp from any to any port = sunrpc keep state @4 pass out quick proto tcp from any to any port 7936:9937 keep state @5 pass out quick proto udp from any to any port 7936:9937 keep state @6 block out all
When running the scheduled backup, over this firewall, it works:
7341:savegrp: newt:/etc/motd succeeded.
The traditional approach basically opens the entire range of TCP ports that EMC NetWorker might use. The alternative approach that we will describe now, tries to remedy this.
First we will snoop on the network traffic, while running a manual EMC NetWorker backup over our firewall (possibly with the traditional rules as explained before enabled):
root@newt:/home/stes# save -s darkstar /etc/motd /etc/motd /etc/ / save: /etc/motd 4 KB 00:00:00 3 files
A snoop of the traffic shows SUN RPC calls (C) and responses (R) to various program numbers such as 100000 (portmapper), 390109 (nsrstat), 390103 (nsrd), 390107 (nsrmmdbd) for the media database, 390105 (nsrindexd) and of course 390104 (nsrmmd) for actually writing the backup to media :
newt -> darkstar RPC C XID=1206590463 PROG=390109 (?) VERS=2 PROC=120
darkstar -> newt RPC R (#27) XID=1206590463 Success
newt -> darkstar RPC C XID=1206602471 PROG=390109 (?) VERS=2 PROC=120
darkstar -> newt RPC R (#61) XID=1206602471 Success
newt -> darkstar RPC C XID=1189825255 PROG=390109 (?) VERS=2 PROC=101
darkstar -> newt RPC R (#69) XID=1189825255 Success
newt -> darkstar RPC C XID=1173048039 PROG=390109 (?) VERS=2 PROC=102
darkstar -> newt RPC R (#73) XID=1173048039 Success
newt -> darkstar RPC C XID=1206619736 PROG=390109 (?) VERS=2 PROC=120
darkstar -> newt RPC R (#115) XID=1206619736 Success
newt -> darkstar RPC C XID=1206632246 PROG=390103 (?) VERS=2 PROC=120
darkstar -> newt RPC R (#149) XID=1206632246 Success
newt -> darkstar RPC C XID=1206640536 PROG=390107 (?) VERS=6 PROC=0
darkstar -> newt RPC R (#189) XID=1206640536 Success
newt -> darkstar RPC C XID=1206646532 PROG=390103 (?) VERS=2 PROC=120
darkstar -> newt RPC R (#223) XID=1206646532 Success
newt -> darkstar RPC C XID=1189869316 PROG=390103 (?) VERS=2 PROC=0
darkstar -> newt RPC R (#231) XID=1189869316 Success
newt -> darkstar RPC C XID=1189863320 PROG=390107 (?) VERS=6 PROC=76
darkstar -> newt RPC R (#235) XID=1189863320 Success
newt -> darkstar RPC C XID=1173092100 PROG=390103 (?) VERS=2 PROC=122
darkstar -> newt RPC R (#239) XID=1173092100 Success
newt -> darkstar RPC C XID=1156314884 PROG=390103 (?) VERS=2 PROC=120
darkstar -> newt RPC R (#243) XID=1156314884 Success
newt -> darkstar RPC C XID=1139537668 PROG=390103 (?) VERS=2 PROC=120
darkstar -> newt RPC R (#277) XID=1139537668 Success
newt -> darkstar RPC C XID=1206558671 PROG=390105 (?) VERS=6 PROC=1
darkstar -> newt RPC R (#317) XID=1206558671 Success
newt -> darkstar RPC C XID=1206543644 PROG=390104 (?) VERS=105 PROC=38
darkstar -> newt RPC R (#329) XID=1206543644 Success
newt -> darkstar RPC C XID=1189781455 PROG=390105 (?) VERS=6 PROC=3
darkstar -> newt RPC R XID=1173004239 Success
newt -> darkstar RPC C XID=1189766428 PROG=390104 (?) VERS=105 PROC=39
newt -> darkstar RPC C XID=1156227023 PROG=390105 (?) VERS=6 PROC=3
darkstar -> newt RPC R XID=1139449807 Success
newt -> darkstar RPC C XID=1122672591 PROG=390105 (?) VERS=6 PROC=3
darkstar -> newt RPC R XID=1105895375 Success
newt -> darkstar RPC C XID=1156211996 PROG=390104 (?) VERS=105 PROC=41
darkstar -> newt RPC R (#385) XID=1156211996 Success
A slight modification of the OpenSolaris IPFilter rules enables the XID extension. We tell the OpenSolaris IPFilter state machinery that it has to check the RPC XID numbers of RPC calls and responses, by setting the rpc option on the keep state :
# allow incoming SSH connections pass out quick proto tcp from any to any port = 22 # allow packets associated to SSH connections pass out quick proto tcp from any port = 22 to any pass out quick proto tcp from any to any port = 111 keep state pass out quick proto udp from any to any port = 111 keep state # enable XID filtering on the states associated to EMC NetWorker pass out quick proto tcp from any to any port 7936:9937 keep state (rpc) pass out quick proto udp from any to any port 7936:9937 keep state (rpc) block out all
The above rules are almost the same as the ones used before. The only difference is the (rpc) keyword between parentheses, to enable tracing RPC calls and responses.
In debug mode, we can now see that OpenSolaris IPFilter is able to trace the communication between EMC NetWorker server and EMC NetWorker client at the RPC level.
Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390103 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 390107 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390103 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 390104 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 0 WAIT-C Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 WAIT-R Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:03:59 gecko ipf: TCP 8998 RPC 390107 ACCEPT
OpenSolaris IPFilter was able to decode for each TCP state the RPC program number and matches the XID of the response to the XID of the call.
For EMC NetWorker restores, the situation is similar. The restore is initiated on the client, and all traffic involves RPC calls between client and server.
root@newt:/home/stes# recover -s darkstar
/home/stes/ not in index
<return> will exit.
Enter directory to browse: /etc
recover: Current working directory is /etc/
recover> add motd
/etc
1 file(s) marked for recovery
recover> recover
recover: Total estimated disk space needed for recover is 1 KB.
Recovering 1 file into its original location
Volumes needed (all on-line):
DISK1 at /home/disk1
Requesting 1 file(s), this may take a while...
./motd
./motd file exists, overwrite (n, y, N, Y) or rename (r, R) [n]? y
overwriting ./motd
Received 1 file(s) from NSR server `darkstar'
Recover completion time: Tue Sep 30 19:37:57 2008
recover> quit
The restore over the firewall generates a bunch of RPC (remote procedure calls) to such services as nsrindexd, nsrmmd, nsrmmdbd.
OpenSolaris IPFilter can report on the actual traffic in debug mode:
Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390103 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8998 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8998 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 8998 RPC 390107 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390103 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8202 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8202 RPC 0 WAIT-C Sep 30 19:33:54 gecko ipf: TCP 8202 RPC 390105 WAIT-R Sep 30 19:33:54 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:33:54 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:33:55 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:33:55 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:33:55 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:34:05 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:34:05 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:34:05 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:34:05 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:34:05 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:34:05 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:34:05 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:34:05 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:34:05 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:34:06 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:34:06 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:34:06 gecko ipf: TCP 8800 RPC 390103 WAIT-R Sep 30 19:34:06 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:34:06 gecko ipf: TCP 9232 RPC 0 WAIT-C Sep 30 19:34:06 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:34:06 gecko ipf: TCP 9232 RPC 0 WAIT-C Sep 30 19:34:06 gecko ipf: TCP 9232 RPC 390104 WAIT-R Sep 30 19:34:06 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:34:10 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:34:12 gecko ipf: TCP 8202 RPC 390105 ACCEPT
Due to the interactive nature of the restore, timing issues can play a role. Indeed it can be seen that many RPC connections over TCP remain open, so the firewall should not time out the TCP state prematurely; there are some known issues where it may help to use the EMC NetWorker environment variable NSR_KEEPALIVE_WAIT.
As already explained before, scheduled backups are a combination of a BSD REXEC like protocol and of a manual backup.
While the latter uses RPC calls, the BSD REXEC protocol is something entirely different (it uses a decimally encoded port number for STDERR), and nsrexec, the variant that EMC NetWorker uses, changes from version to version (it is derived from rexec).
For our backup server, running EMC NetWorker version 7.4 SP3, we have the following ports registered :
bash-3.00# rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
390436 1 tcp 8743
390435 1 tcp 9573
390113 1 tcp 7937 nsrexecd
390103 2 tcp 8800 nsrd
390109 2 tcp 8800 nsrstat
390110 1 tcp 8800 nsrjbd
390120 1 tcp 8800
390109 2 udp 9001 nsrstat
390107 5 tcp 8998 nsrmmdbd
390107 6 tcp 8998 nsrmmdbd
390433 1 tcp 8455 nsrjobd
390105 5 tcp 8202 nsrindexd
390105 6 tcp 8202 nsrindexd
390104 105 tcp 9232 nsrmmd
When we run a scheduled backup,
bash-3.00# savegrp -l 0 -c newt
The scheduled backup works with the alternative approach, and the communication between server and client involves some interesting TCP ports :
Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 9388 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:26 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 9388 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 9388 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 9388 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 9388 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 9388 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 9891 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7937 RPC 390113 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390109 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390103 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8998 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8998 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 8998 RPC 390107 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390103 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 9232 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 9232 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8202 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 7938 RPC 100000 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8202 RPC 0 WAIT-C Sep 30 19:51:29 gecko ipf: TCP 8202 RPC 390105 WAIT-R Sep 30 19:51:29 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:51:29 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 WAIT-R Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8800 RPC 390103 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9891 RPC 0 WAIT-C Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9891 RPC 390105 WAIT-R Sep 30 19:51:30 gecko ipf: TCP 9891 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9232 RPC 390104 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8202 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9891 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8998 RPC 390107 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9891 RPC 390105 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 8800 RPC 390109 ACCEPT Sep 30 19:51:30 gecko ipf: TCP 9891 RPC 390105 ACCEPT
From the debug output, it can be observed that some TCP ports 9891 are not in the rpcinfo output, but that these TCP ports are dynamically registered for EMC NetWorker RPC services such as 390105.