Grid FTP configuration issues between cheaha.uabgrid and cheaha.ac

The problem has to do with with some firewall issues and inconsistent configuration settings.

1) the firewall entries for cheaha.ac's public IP address don't accept connections from inside the ENG network.

Grid ftp use passive mode transfers on the data channels. When a file transfer is initiated from a client, the server host (the gsiftp:// target, cheaha.ac in this case) a passive mode connection is initiated. This means the server tells the client on which port it is listening and then the client connects to that port to transfer the data.

In the case of the above command, cheaha.ac is configured to advertise its public IP address to clients (/etc/xinetd.d/grid-ftp on cheaha.ac). When grid ftp is initiated from cheaha.uabgrid, it is told to connect to cheaha.ac's public IP address and TCP port 45003 (175*256+203):

debug: response from gsiftp://cheaha.ac.uab.edu/etc/hosts:
227 Entering Passive Mode (138,26,84,126,175,203)

debug: sending command:
RETR /etc/hosts

Tracing the TCP connection on the wire shows the SYN packet from (10.26.84.72) cheaha.uabgrid to 138.26.84.126:45003 (cheaha.ac public IP) but cheaha.ac never responds.

This is a firewall problem. For some reason, cheaha.ac is not receiving requests to the 45000:45999 port range routed to it from inside the ENG network. (Or cheaha.ac response to the SYN packet is blocked) This works from outside the ENG network since I can transfer files from stage.uabgrid (and other external hosts) to cheaha.ac.

2) the grid ftp config on cheaha.uabgrid needs some updates.

The grid ftp server configuration on cheaha.uabgrid in /etc/xinetd.d/ is not complete. It currently doesn't advertise it's public IP address for data transfers. This is needed so that external clients can connect to the proper IP address (as advertised in the FTP control stream above). This is why the transfer from cheaha.ac to cheaha.uabgrid works. cheaha.uabgrid is advertising its ENG IP as the passive end-point address and cheaha.ac can connect to it.

We need to apply the following patch to cheaha.uabgrid /etc/xinetd.

patch -d /etc/xinetd.d gsiftp << EOF
6a7
>     env             += GLOBUS_TCP_SOURCE_RANGE=45000,45999
10c11
<     server_args     = -i -1
---
>     server_args     = -i -1  -data-interface 138.26.84.72
EOF

The upshot of this will likely be that transfers from cheaha.ac to cheaha.uabgrid will stop working until the firewall issue described above is fixed.

The problem was determined to be a conflict between the need to advertise the external IP address for the host in the GridFTP control channel (via --data-interface in the xinetd config file and the ability of the network firewall to route internal and external traffic identically.

Two solutions were proposed

1. Fix the firewall routing problem 1. Add a virtual IP address to the hosts so that one IP can serve internal requests (between hosts behind the firewall) and one can serve external hosts. This would allow distinct GridFTP services to serve specific IP addresses and the --data-interface could advertise the correct IP.

Resolution of this ticket is awaiting selection and implementation of one of the proposed fixes.

It appears resolving this in the firewall with transparent routing for private and public IP addresses is not an option.

With the current network configuration, this leaves the option for assigning a distinct private IP for servicing public IP address requests. This would enable a distinct grid ftp configuration that could tie clients to the correct IP using the --data-interface option. That is, private-IP1 would be advertised by an instance of grid ftp listening on private-IP1 and public-IP would be advertised by an instance of grid ftp listening on private-IP2. It's not clear if this would work in the end given some questions surrounding how name resolution occurs on the private network.

A question was raised whether the FQDN in the --data-interface should produce a FQDN+decimal_port in the control channel's response. That is producing:

227 Entering Passive Mode (cheaha,uabgrid,uab,edu,175,203)

instead of:

227 Entering Passive Mode (138,26,84,126,175,203)

Determining if only dotted-decimal values are allowed in the control channel advertisement is being investigated.

GridFTP is an extension of the original FTP protocol described in RFC959. The GridFTP Protocol Specification v1 is described in the OGF document GDF.20 as a set of extentions to the base protocol, and relies on RFC959 to describe the common foundations.

The control channel commands we're discussing above are defined in RFC959. The endpoint connections for data transfers are identified by a 'host-port' identifier. The syntax rules for the host-port are:

            <host-port> ::= <host-number>,<port-number>
            <host-number> ::= <number>,<number>,<number>,<number>
            <port-number> ::= <number>,<number>
            <number> ::= any decimal integer 1 through 255

That is, the syntax must be decimal, comma separated notation of the hostIP+portNumber. FQDNs are not allowed.

This eliminates the option of sending an FQDN to the client as an endpoint identifier.

For reference, OGF document GDF.47 describes proposed improvements for GridFTP v2. I'm not sure what the status of GridFTP v2 is, though. A background of the development of GridFTP is available in the 'Introduction' section of GDF.20

Given that olympus.cis.uab.edu and everest.cis.uab.edu are both configured to support Globus and apparently also reside behind a common firewall, I've taken a look at their configuration to see if it sheds some light.

Both of these hosts identify themselves with their public IP address on their network interfaces. Therefore, they will advertise this IP address to all clients and directly accept the connections for this IP.

I've sent an email to clusterhelp@cis.uab.edu to better understand their network and firewall configuration.

I sent an email to gridftp-user@globus.org (archive link not yet available) to ask for configuration feed back on this scenario. The recommendation was that configuring dedicated GridFTP servers to listen on specific interfaces is the best approach.

I think we should move forward with this configuration.

We need to confirm two configuration aspects to determine if this will work:

1. hostname resolution.
2. Xinetd configuration for distinct interfaces

In order to configure GridFTP to work for internal and external hosts, we will need two distinct ENG-net internal IP addresses for any cluster on the grid: one to receive traffic for the publicIP and one to receive traffic for the privateIP.

The existing private IP for cheaha.ac and cheaha.uabgrid, which accept traffic from external clients, can remain configured as they are. This interface will have a GridFTP server configured that listens at this IP and exposes the publicIP via the --data-interface argument.

The second, pseudo-interface will have privateIP2 and have a GridFTP server configured to listen at this IP and expose it's privateIP2 via --data-interface.

Each cluster will then need to identify the peer clusters by their privateIP2 via an entry in /etc/hosts. This should enable the clusters to talk to each other.

There are some downsides, all other hosts that want to talk directly to the grid interfaces from the ENG network will need these /etc/hosts entries.

And after writing all the above, this suggests we may need distinct Globus services for each IP. We'll need to investigate this. If it's true, we'll be better off pursuing a new solution.

The Xinetd configuration should be straight forward. Either we can specify the alt IP in the xinetd config of we run a dedicated GridFTP server statically on the privateIP2.

Added mhanby to ticket CC. Seems his email got removed at some update along the way.

It looks like mhanby was removed when I changed the ticket owner to jpr on 2008-10-20. Apparently, changing the ticket owner does not add the email of the prior owner to the CC list.

We will need to make sure the CC list includes the owner of the ticket or remember to add the owners address to the CC list if an ownership change is made.

Replying to jpr@uab.edu:

Added mhanby to ticket CC. Seems his email got removed at some update along the way.