OpenSolaris IPFilter rules for EMC NetWorker

David Stes
email: stes@pandora.be

September 24, 2008

Abstract:

OpenSolaris IPFilter makes it possible to do backups with EMC NetWorker over a Solaris firewall. The firewall is a UNIX router that is filtering the SUN RPC (remote procedure calls) of EMC NetWorker. The actual rules to configure the firewall are discussed here, with practical experiences on setting up the software. First, the traditional approach is implemented, where a range of ports is opened, as documented by the nsrports tool of EMC NetWorker. But in addition to this, we also use an alternative approach, using XID (transaction ID) filtering technology, as described in previous papers. Using this alternative approach, only those ports that are associated to specific SUN RPC program numbers, are opened.

EMC NetWorker backup server

The following ip addresses are used:

172.16.0.8      gecko 
192.168.0.8     gecko-bge0
172.16.0.100    newt
192.168.0.5     darkstar

Our backup server is a Linux system called darkstar. It is running EMC NetWorker version 7.4 SP3.

# nsradmin -s darkstar -v1 -p nsrexec
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin>print
                        type: NSRLA;
                        name: darkstar;
                     version: EMC NetWorker 7.4.3.Build.569 08/28/08;

The client software of EMC NetWorker 7.4 SP3 registers the following SUN RPC ports, as can be shown using our nwrpcinfo tool :

bash-3.00# nwrpcinfo -p
   program vers proto   port
    100000    2   tcp   7938  portmapper
    100000    2   udp   7938  portmapper
    390436    1   tcp   8194
    390435    1   tcp   9024
    390113    1   tcp   7937  nsrexecd

The 390435 and 390436 are unnamed, and not further discussed by EMC, but these programs were introduced in EMC NetWorker 7.3 (see our paper of 2006 on netfilter firewalling); they are related to authserver. On our backup server, we use the default auth methods, although that in practice it may be preferable to use oldauth only :

nsradmin> update auth methods: "0.0.0.0/0,nsrauth/oldauth"
                auth methods: "0.0.0.0/0,nsrauth/oldauth";
Update? y

The nwrpcinfo tool is a modification of the standard rpcinfo. It is compiled from the Linux libc C sources but with a modification to query the EMC portmapper, by using PMAPPORT 7938 instead of 111 :

/usr/include/rpc/pmap_prot.h:#define PMAPPORT           ((u_short)111)
/usr/include/rpc/pmap_prot.h:#define PMAPPORT           ((u_short)7938)
/* textdomain (_libc_intl_domainname); don't use gettext */

The TCP and UDP port numbers that EMC NetWorker uses are dynamically allocated from a range of ports. For example, for each (enabled) tape drive, a different nsrmmd process is created with a different TCP address, and EMC NetWorker will listen on that port for data that needs to be written to that specific tape unit.

After labeling a disk device, so that we have a volume to do backups, the following SUN RPC program numbers are registered on our backup server :

bash-3.00# nwrpcinfo -p
   program vers proto   port
    100000    2   tcp   7938  portmapper
    100000    2   udp   7938  portmapper
    390436    1   tcp   8194
    390435    1   tcp   9024
    390113    1   tcp   7937  nsrexecd
    390103    2   tcp   9141  nsrd
    390109    2   tcp   9141  nsrstat
    390110    1   tcp   9141  nsrjbd
    390120    1   tcp   9141
    390109    2   udp   8865  nsrstat
    390107    5   tcp   9389  nsrmmdbd
    390107    6   tcp   9389  nsrmmdbd
    390433    1   tcp   8824  nsrjobd
    390105    5   tcp   8806  nsrindexd
    390105    6   tcp   8806  nsrindexd
    390104  105   tcp   8694  nsrmmd

In terms of service port range, we use the default range :

# nsrports -s darkstar
Service ports: 7937-9936 
Connection ports: 0-0

The SUN RPC program numbers for EMC NetWorker (formerly Legato) used to be managed by SUN, it is nowadays available at:

http://www.nfsv4-editor.org/rpc-numbers-1831bis.txt

EMC NetWorker client

The backup client is a different Linux system called newt. It is running EMC NetWorker version 6.1.3, which was at the time still a Legato product, so our client is Legato NetWorker 6.1.3.

Unfortunately, the EMC NetWorker 7.4 SP3 software prints out the following error when we try to make a command-line connection to the agent :

bash-3.00# nsradmin -s newt -p nsrexec -v1
39078:nsradmin: RPC error: Program not registered

The EMC NetWorker 7.4 SP3 software is complaining about the fact that there is no SUN RPC 390436 running on newt :

    darkstar -> newt         PORTMAP C GETPORT prog=390436 (?) vers=1 proto=TCP
    darkstar -> newt         PORTMAP C GETPORT prog=390436 (?) vers=1 proto=TCP

With the old nsradmin executable, we can still make a connection:

# nsradmin613 -s newt -p nsrexec -v1
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin613> print
                        type: NSRLA;
                        name: newt;
                     version: Legato NetWorker 6.1.3.Build.428 11/21/02;

The old executable is in some sense more powerful than the new one, since it also allows us to connect to the EMC NetWorker 7.4 software. This is due of course to the fact that the old executable doesn't issue a port request to the 390436 program. It could be argued that the behavior of the new nsradmin executable could be improved, so that is able to connect to older agents.

The client is running the following SUN RPC services:

# nwrpcinfo -p newt
    program vers proto   port
    100000    2   tcp   7938  portmapper
    100000    2   udp   7938  portmapper
    390113    1   tcp   7937  nsrexecd

For the service port range, newt is also using the default settings:

# nsrports -s newt
Service ports: 7937-9936 
Connection ports: 10001-30000

Multi-homed OpenSolaris IPFilter server

The hostname of our router is gecko, it's a PCI-e system with two gigabit interfaces :

# dladm show-link
LINK        CLASS    MTU    STATE    OVER
e1000g0     phys     1500   up       --
bge0        phys     1500   up       --

One interface is in the same subnet as the EMC NetWorker backup server, and the other interface is in the subnet of the EMC NetWorker client :

stes@gecko:~# netstat -rn

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
172.16.0.0           172.16.0.8           U         1         18 e1000g0   
192.168.0.0          192.168.0.8          U         1         11 bge0      
127.0.0.1            127.0.0.1            UH        1          0 lo0

The operating system is OpenSolaris 2008.05 :

stes@gecko:# uname -a
SunOS gecko 5.11 snv_86 i86pc i386 i86pc Solaris

Because this system acts as a router, we enabled IP forwarding, and disabled the automatic configuration of network interfaces.

stes@gecko:# svcs -a | grep forward
online          9:35:35 svc:/network/ipv4-forwarding:default

stes@gecko:~# svcs -a | grep physical
disabled       17:49:32 svc:/network/physical:nwam
online         17:49:37 svc:/network/physical:default

By default, this system comes with version 4.1.9 of OpenSolaris IPFilter. However, we uninstalled the standard packages SUNWipfr and SUNWipfu (using the command pkg uninstall SUNWipf). Next, we compiled ourselves the latest version of OpenSolaris IPFilter, so the software that is described in this paper is version 5.0.4, which is a different branch of the software :

stes@gecko:~# pkginfo -l ipfx
   PKGINST:  ipfx
      NAME:  IP Filter (64-bit)
  CATEGORY:  system
      ARCH:  i386
   VERSION:  5.0.4
   BASEDIR:  /
    VENDOR:  Darren Reed
      DESC:  This package contains tools for building a firewall
  INSTDATE:  Sep 21 2008 04:08
     EMAIL:  darrenr@pobox.com

The compilation itself works as follows on OpenSolaris :

svccfg export network/ipfilter > /tmp/ipfilter.def
cp /lib/svc/method/ipfilter /tmp
svcadm disable ipfilter
pkgrm ipf ipfx
make solaris
cd SunOS5;make pkg
make package
svcadm enable ipfilter
# ipf -E (if necessary, enable should do this)

OpenSolaris IPFilter Configuration (traditional approach)

The first approach to configure the OpenSolaris IPFilter is to allow TCP packets of the EMC NetWorker port range to pass, and we can write some rules for this. These rules use stateful inspection of the TCP sessions i.e., we request that OpenSolaris IPFilter stores the TCP state of each connection.

Check the firewall:

# ipf -V
ipf: IP Filter: v5.0.4 (648)
Kernel: IP Filter: v5.0.4               
Running: no
Log Flags: 0 = none set
Default: pass all, Logging: available
Active list: 0
Feature mask: 0x107

Because OpenSolaris IPFilter is not yet Running, we enable it and ask it to log blocked packets:

stes@gecko:~# ipf -E
stes@gecko:~# ipf -l blocked
stes@gecko:~# ipf -V
ipf: IP Filter: v5.0.4 (648)
Kernel: IP Filter: v5.0.4               
Running: yes
Log Flags: 0x20000000 = block
Default: pass all, Logging: available
Active list: 0
Feature mask: 0x107

Flush (empty) the list of firewall rules, then load the firewall rules, allowing SSH over the firewall and opening the ports in the range of 7937 to 9936.

stes@gecko:/etc/ipf# ipf -Fo
stes@gecko:/etc/ipf# cat /etc/ipf/ipf.nsr
pass out quick proto tcp from any to any port = ssh keep state
pass out quick proto tcp from any to any port 7936:9937 keep state
pass out quick proto udp from any to any port 7936:9937 keep state
block out all

stes@gecko:/etc/ipf# ipf -o -f /etc/ipf/ipf.nsr

stes@gecko:/etc/ipf# ipfstat -on
@1 pass out quick proto tcp from any to any port = ssh keep state
@2 pass out quick proto tcp from any to any port 7936:9937 keep state
@3 pass out quick proto udp from any to any port 7936:9937 keep state
@4 block out all

After enabling this set of rules, it can be observed that ping doesn't work any longer between the two Linux systems, and that we can still run SSH over the firewall. The number of matches for rules can be displayed as follows:

stes@gecko:/etc/ipf# ipfstat -ho
1 pass out quick proto tcp from any to any port = ssh keep state
0 pass out quick proto tcp from any to any port 7936:9937 keep state
0 pass out quick proto udp from any to any port 7936:9937 keep state
2 block out all

Manual and Scheduled Backups (traditional approach)

As discussed in previous papers on EMC NetWorker, from a point of view of network protocols, EMC NetWorker uses two different protocols for manual and scheduled backups. We will first investigate here how these backups work with our firewall (that is configured as is traditionally done, by opening a range of ports).

To simplify things, it could be stated that basically, EMC NetWorker uses SUN RPC to transfer data from the client to the server during manual (client initiated) backups.

Again, simplifying things, it can be stated that essentially, EMC NetWorker uses a BSD rexec derived protocol, for the server to contact the client during a scheduled backup, where the server requests remote command execution of the command to run a manual backup.

This is a simplification, and many EMC NetWorker products heavily use SUN RPC for such things as initiating VSS or other PowerSnap snapshots etc. but the basic distinction between manual and scheduled backup is essential, in terms of network protocols.

First, we try the manual (client initiated) backup over our firewall:

root@newt:/# save -s darkstar /etc/motd
/etc/motd
/etc/
/

save: /etc/motd  4 KB 00:00:01      3 files

The backup works and it can be observed that it uses various TCP sessions:

stes@gecko:/etc/ipf# ipfstat -ho
1 pass out quick proto tcp from any to any port = ssh keep state
16 pass out quick proto tcp from any to any port 7936:9937 keep state
0 pass out quick proto udp from any to any port 7936:9937 keep state
13 block out all

The case of scheduled backups is usually much more complicated with EMC NetWorker, as it involves a variety of protocols, depending on the client (and server) version.

Without firewall, it works fine:

bash-3.00# savegrp -v -c newt 
32451:savegrp: newt:/etc/motd                            level=incr
7236:savegrp: Group will not limit job parallelism
32493:savegrp: newt:probe                                    started
savefs -s darkstar -c newt -g Default -p -l full -R -v -F /etc/motd
savegrp:Default * newt:Probe  See the file /nsr/tmp/sg/Default/sso.newt.6ZlZC0 for output of save command.
7340:savegrp: newt:probe succeeded.
newt:/etc/motd                     level=full, dn=0, mx=1, vers=ssbrowse, p=12
32494:savegrp: newt:/etc/motd                                started
save -s darkstar -g Default -LL -m newt -l full -W 78 -N /etc/motd /etc/motd
savegrp:Default * newt:/etc/motd  See the file /nsr/tmp/sg/Default/sso.newt.8KD4nj for output of save command.

Unfortunately, when we enable the firewall, the scheduled backup hangs. It can be seen that the OpenSolaris IPFilter is blocking the following packets:

# ipmon -a

24/09/2008 20:30:09.734429 e1000g0 @0:4 b 192.168.0.5,2687 -> 172.16.0.100,111 PR tcp len 20 60 -S OUT

The command ipmon shows some requests to the (SUN) portmapper 111, and this makes some sense, in the context of the EMC NetWorker protocols, so we add a rule specifically for this.

stes@gecko:/etc/ipf# cat /etc/ipf/ipf.nsr
pass out quick proto tcp from any to any port = ssh keep state
pass out quick proto tcp from any to any port = 111 keep state
pass out quick proto udp from any to any port = 111 keep state
pass out quick proto tcp from any to any port 7936:9937 keep state
pass out quick proto udp from any to any port 7936:9937 keep state
block out all

stes@gecko:/etc/ipf# ipf -o -f /etc/ipf/ipf.nsr
stes@gecko:/etc/ipf# ipfstat -on
@1 pass out quick proto tcp from any to any port = ssh keep state
@2 pass out quick proto tcp from any to any port = sunrpc keep state
@3 pass out quick proto udp from any to any port = sunrpc keep state
@4 pass out quick proto tcp from any to any port 7936:9937 keep state
@5 pass out quick proto udp from any to any port 7936:9937 keep state
@6 block out all

When running the scheduled backup, over this firewall, it works:

7341:savegrp: newt:/etc/motd succeeded.

The traditional approach basically opens the entire range of TCP ports that EMC NetWorker might use. The alternative approach that we will describe now, tries to remedy this.

Manual EMC NetWorker Backup (alternative approach)

First we will snoop on the network traffic, while running a manual EMC NetWorker backup over our firewall (possibly with the traditional rules as explained before enabled):

root@newt:/home/stes# save -s darkstar /etc/motd
/etc/motd
/etc/
/

save: /etc/motd  4 KB 00:00:00      3 files

A snoop of the traffic shows SUN RPC calls (C) and responses (R) to various program numbers such as 100000 (portmapper), 390109 (nsrstat), 390103 (nsrd), 390107 (nsrmmdbd) for the media database, 390105 (nsrindexd) and of course 390104 (nsrmmd) for actually writing the backup to media :

        newt -> darkstar     RPC C XID=1206590463 PROG=390109 (?) VERS=2 PROC=120
    darkstar -> newt         RPC R (#27) XID=1206590463 Success
        newt -> darkstar     RPC C XID=1206602471 PROG=390109 (?) VERS=2 PROC=120
    darkstar -> newt         RPC R (#61) XID=1206602471 Success
        newt -> darkstar     RPC C XID=1189825255 PROG=390109 (?) VERS=2 PROC=101
    darkstar -> newt         RPC R (#69) XID=1189825255 Success
        newt -> darkstar     RPC C XID=1173048039 PROG=390109 (?) VERS=2 PROC=102
    darkstar -> newt         RPC R (#73) XID=1173048039 Success
        newt -> darkstar     RPC C XID=1206619736 PROG=390109 (?) VERS=2 PROC=120
    darkstar -> newt         RPC R (#115) XID=1206619736 Success
        newt -> darkstar     RPC C XID=1206632246 PROG=390103 (?) VERS=2 PROC=120
    darkstar -> newt         RPC R (#149) XID=1206632246 Success
        newt -> darkstar     RPC C XID=1206640536 PROG=390107 (?) VERS=6 PROC=0
    darkstar -> newt         RPC R (#189) XID=1206640536 Success
        newt -> darkstar     RPC C XID=1206646532 PROG=390103 (?) VERS=2 PROC=120
    darkstar -> newt         RPC R (#223) XID=1206646532 Success
        newt -> darkstar     RPC C XID=1189869316 PROG=390103 (?) VERS=2 PROC=0
    darkstar -> newt         RPC R (#231) XID=1189869316 Success
        newt -> darkstar     RPC C XID=1189863320 PROG=390107 (?) VERS=6 PROC=76
    darkstar -> newt         RPC R (#235) XID=1189863320 Success
        newt -> darkstar     RPC C XID=1173092100 PROG=390103 (?) VERS=2 PROC=122
    darkstar -> newt         RPC R (#239) XID=1173092100 Success
        newt -> darkstar     RPC C XID=1156314884 PROG=390103 (?) VERS=2 PROC=120
    darkstar -> newt         RPC R (#243) XID=1156314884 Success
        newt -> darkstar     RPC C XID=1139537668 PROG=390103 (?) VERS=2 PROC=120
    darkstar -> newt         RPC R (#277) XID=1139537668 Success
        newt -> darkstar     RPC C XID=1206558671 PROG=390105 (?) VERS=6 PROC=1
    darkstar -> newt         RPC R (#317) XID=1206558671 Success
        newt -> darkstar     RPC C XID=1206543644 PROG=390104 (?) VERS=105 PROC=38
    darkstar -> newt         RPC R (#329) XID=1206543644 Success
        newt -> darkstar     RPC C XID=1189781455 PROG=390105 (?) VERS=6 PROC=3
    darkstar -> newt         RPC R XID=1173004239 Success
        newt -> darkstar     RPC C XID=1189766428 PROG=390104 (?) VERS=105 PROC=39
        newt -> darkstar     RPC C XID=1156227023 PROG=390105 (?) VERS=6 PROC=3
    darkstar -> newt         RPC R XID=1139449807 Success
        newt -> darkstar     RPC C XID=1122672591 PROG=390105 (?) VERS=6 PROC=3
    darkstar -> newt         RPC R XID=1105895375 Success
        newt -> darkstar     RPC C XID=1156211996 PROG=390104 (?) VERS=105 PROC=41
    darkstar -> newt         RPC R (#385) XID=1156211996 Success

A slight modification of the OpenSolaris IPFilter rules enables the XID extension. We tell the OpenSolaris IPFilter state machinery that it has to check the RPC XID numbers of RPC calls and responses, by setting the rpc option on the keep state :

# allow incoming SSH connections
pass out quick proto tcp from any to any port = 22
# allow packets associated to SSH connections
pass out quick proto tcp from any port = 22 to any

pass out quick proto tcp from any to any port = 111 keep state
pass out quick proto udp from any to any port = 111 keep state

# enable XID filtering on the states associated to EMC NetWorker
pass out quick proto tcp from any to any port 7936:9937 keep state (rpc)
pass out quick proto udp from any to any port 7936:9937 keep state (rpc)
block out all

The above rules are almost the same as the ones used before. The only difference is the (rpc) keyword between parentheses, to enable tracing RPC calls and responses.

In debug mode, we can now see that OpenSolaris IPFilter is able to trace the communication between EMC NetWorker server and EMC NetWorker client at the RPC level.

Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390103 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 390107 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390103 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 390104 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 0 WAIT-C
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 WAIT-R
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:03:59 gecko ipf:  TCP 8998 RPC 390107 ACCEPT

OpenSolaris IPFilter was able to decode for each TCP state the RPC program number and matches the XID of the response to the XID of the call.

EMC NetWorker Restore (alternative approach)

For EMC NetWorker restores, the situation is similar. The restore is initiated on the client, and all traffic involves RPC calls between client and server.

root@newt:/home/stes# recover -s darkstar
/home/stes/ not in index
<return> will exit.
Enter directory to browse: /etc
recover: Current working directory is /etc/
recover> add motd
/etc
1 file(s) marked for recovery
recover> recover
recover: Total estimated disk space needed for recover is 1 KB.
Recovering 1 file into its original location
Volumes needed (all on-line):
        DISK1 at /home/disk1
Requesting 1 file(s), this may take a while...
./motd
./motd file exists, overwrite (n, y, N, Y) or rename (r, R) [n]? y
overwriting ./motd
Received 1 file(s) from NSR server `darkstar'
Recover completion time: Tue Sep 30 19:37:57 2008
recover> quit

The restore over the firewall generates a bunch of RPC (remote procedure calls) to such services as nsrindexd, nsrmmd, nsrmmdbd.

OpenSolaris IPFilter can report on the actual traffic in debug mode:

Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390103 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8998 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8998 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 8998 RPC 390107 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390103 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8202 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8202 RPC 0 WAIT-C
Sep 30 19:33:54 gecko ipf:  TCP 8202 RPC 390105 WAIT-R
Sep 30 19:33:54 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:33:54 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:33:55 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:33:55 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:33:55 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:34:05 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:34:05 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:34:05 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:34:05 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:34:05 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:34:05 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:34:05 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:34:05 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:34:05 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:34:06 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:34:06 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:34:06 gecko ipf:  TCP 8800 RPC 390103 WAIT-R
Sep 30 19:34:06 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:34:06 gecko ipf:  TCP 9232 RPC 0 WAIT-C
Sep 30 19:34:06 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:34:06 gecko ipf:  TCP 9232 RPC 0 WAIT-C
Sep 30 19:34:06 gecko ipf:  TCP 9232 RPC 390104 WAIT-R
Sep 30 19:34:06 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:34:10 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:34:12 gecko ipf:  TCP 8202 RPC 390105 ACCEPT

Due to the interactive nature of the restore, timing issues can play a role. Indeed it can be seen that many RPC connections over TCP remain open, so the firewall should not time out the TCP state prematurely; there are some known issues where it may help to use the EMC NetWorker environment variable NSR_KEEPALIVE_WAIT.

EMC NetWorker Scheduled Backup (alternative approach)

As already explained before, scheduled backups are a combination of a BSD REXEC like protocol and of a manual backup.

While the latter uses RPC calls, the BSD REXEC protocol is something entirely different (it uses a decimally encoded port number for STDERR), and nsrexec, the variant that EMC NetWorker uses, changes from version to version (it is derived from rexec).

For our backup server, running EMC NetWorker version 7.4 SP3, we have the following ports registered :

bash-3.00# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    390436    1   tcp   8743
    390435    1   tcp   9573
    390113    1   tcp   7937  nsrexecd
    390103    2   tcp   8800  nsrd
    390109    2   tcp   8800  nsrstat
    390110    1   tcp   8800  nsrjbd
    390120    1   tcp   8800
    390109    2   udp   9001  nsrstat
    390107    5   tcp   8998  nsrmmdbd
    390107    6   tcp   8998  nsrmmdbd
    390433    1   tcp   8455  nsrjobd
    390105    5   tcp   8202  nsrindexd
    390105    6   tcp   8202  nsrindexd
    390104  105   tcp   9232  nsrmmd

When we run a scheduled backup,

bash-3.00# savegrp -l 0 -c newt

The scheduled backup works with the alternative approach, and the communication between server and client involves some interesting TCP ports :

Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 9388 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:26 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 9388 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 9388 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 9388 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 9388 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 9388 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 9891 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7937 RPC 390113 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390109 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390103 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8998 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8998 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 8998 RPC 390107 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390103 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 9232 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 9232 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8202 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 7938 RPC 100000 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8202 RPC 0 WAIT-C
Sep 30 19:51:29 gecko ipf:  TCP 8202 RPC 390105 WAIT-R
Sep 30 19:51:29 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:51:29 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 WAIT-R
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8800 RPC 390103 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9891 RPC 0 WAIT-C
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9891 RPC 390105 WAIT-R
Sep 30 19:51:30 gecko ipf:  TCP 9891 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9232 RPC 390104 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8202 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9891 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8998 RPC 390107 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9891 RPC 390105 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 8800 RPC 390109 ACCEPT
Sep 30 19:51:30 gecko ipf:  TCP 9891 RPC 390105 ACCEPT

From the debug output, it can be observed that some TCP ports 9891 are not in the rpcinfo output, but that these TCP ports are dynamically registered for EMC NetWorker RPC services such as 390105.



David Stes
2008-09-30