Transferring Files

From GridPP Wiki
Revision as of 13:04, 27 October 2015 by Jens Sha2 Jensen c7e29f120b (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

This page describes how to transfer files into, out of, or across the GridPP infrastructure. Several tools are available; some experiments use their own tools (not documented here) usually layered on top of the lower level tools.

File Transfer Service

The file transfer service is based on the EMI FTS3. CERN runs an FTS3 service; the [RAL Tier 1] runs an FTS service as well albeit without a web front end. These instructions are for the use of the RAL FTS.

File Transfer Jobs

File transfer requests are submitted to FTS in jobs, each job containing between 1 and 2000 transfers. At the time of the submission, a proxy is delegated (see the Proxies section below) which will manage the transfers within the job, so the proxy lifetime must be long enough to accommodate all the transfers.

Important FTS switches (for fts-transfer-submit)

  • -s - service endpoint: use https://lcgfts3.gridpp.rl.ac.uk:8443/
  • -e - expiration time: note that the documentation says it's in minutes but it is in seconds!
  • -K - for comparing checksums between source and destination (usually ADLER32)
  • -o - for overwriting destination files (use with caution)
  • -r - reuse the channel - particularly important for transferring lots of (small) files
  • --retry - the retry count; by default FTS will try only once as retries can sometimes be counterproductive (wasting time transferring a file that cannot be transferred)

Other things to remember:

  • If overwrite is off, transferring an existing file is an error (it will count as a failed transfer)
  • FTS will, oddly, refuse to transfer a zero length file. Not only that, but the transfer will count as a failed transfer.
  • The transfers will only use the proxy with which they were submitted; not any later (or fresher) proxies. See the Proxies section below.
  • Source and destination must be full SURLs or TURLs (if this doesn't make sense, think of full URLs).

Proxies

  • FTS will do a delegation based on the local proxy the user holds (grid or voms proxy) when:
    • the user runs a delegating command such as fts-delegation-init or fts-transfer-submit
    • AND the local proxy has a lifetime more than one hour
    • AND the server does not already hold any proxy for the user (DN) with a lifetime of more than four hours
  • If the delegation is attempted and the local proxy lifetime is less than one hour then the delegation will fail.
  • When a job is submitted, the transfers will use the proxy available at the time (delegated with the submission or the one on the server if validity is >4hrs) - if this proxy expires, every transfer will fail. There is no renewal of proxies (Ã la MyProxy).

When using plain old grid proxies, the best approach is thus to create a proxy with as long a lifetime as possible, delegate it to the server and then forget about it.

Now the thick plottens. Suppose we want to use a VOMS proxy. VOMS attributes (embedded in the voms proxy) have a maximal lifetime (by configuration/policy) of 24 hours, so the complete proxy can have a lifetime of at most 24 hours.

It follows that one should:

  1. . Get as long a proxy as possible (e.g. 24 hrs) and delegate it to FTS
  2. . Submit as many transfers as can comfortably fit into that lifetime minus four hours (so 20 hrs) - however, transfers can and do continue for the full lifetime of the proxy
  3. . Keep an eye on the progress of the transfers (to submit more/fewer or even cancel as needed)
  4. . When there is less than four hours left on the server's proxy, and just prior to a new job is being submitted, create a new proxy.

Of course if one creates proxies manually it becomes inconvenient to create new proxies every twenty-hours-and-one-minute. It may make sense then to create a proxy in the morning (say, at 9am) with a lifetime of 11.5 hours, so it will expire at 20:30; and then create a new proxy at 17:00 with a lifetime of 19.5 hours, so it expires at 12:30 the following day - and then create a new proxy at 09:00 the following day. This way, once always has a fresh VOMS proxy to delegate and run the transfers. The downside is to remember to do it and one also has to do it during the weekend.

Globus Online

TBA

xroot

TBA