Long running jobs using myproxy

*** THIS PAGE CONTAINS OBSOLETE INFORMATION ***

This is a solution to a well-known problem whereby long glite file transfers or other long operations fail if they outlive the lifetime of the voms proxy at time of submission.

Introduction and context

A decent explanation of the workings of long term proxies is given here: http://www.gridpp.ac.uk/deployment/users/myproxy.html

The user guide also contains much contextual information on proxy delegation: https://edms.cern.ch/file/722398/1.4/gLite-3-UserGuide.pdf

Please read and understand those sources before trying this procedure.

Preliminary steps

Note that there must be a valid proxy on the UI, created with grid-proxy-init or voms-proxy-init, to successfully interact with a long-term proxy on the MyProxy server. So the first step is to create a voms proxy for your vo:

$ voms-proxy-init -voms <vo>

You can query your proxy as follows:

$ voms-proxy-info -all

And it can be destroyed as so

$ voms-proxy-destroy

Proxy Renewal Concepts

Proxies created as described in the previous section pose a problem: if a job does not finish before the expiration time of the proxy used to submit it, is aborted. This can easily happen, for example, if the job takes a very long time to execute, or if it stays in a queue for a long time. The easiest solution to the problem would be to use very long-lived proxies, but at the expense of an increased security risk. To overcome this limitation, a proxy credential repository system is used, which allows the user to create and store a long-term proxy in a dedicated server (a MyProxy server). The WMS will then be able to use this long-term proxy to periodically renew the proxy for a submitted job before it expires and until the job ends (or the long-term proxy expires). A UI may have a default server defined in the MYPROXY SERVER environment variable. Management of the proxy renewal functionality is available via the myproxy commands.

Long-term proxy lifecycle

To create and store a long-term proxy:

$ myproxy-init -s <myproxy_server> -d -n

where -s <myproxy server> specifies the hostname of the machine where a MyProxy Server runs, the -d option instructs the server to associate the user DN to the proxy, and the -n option avoids the use of a passphrase to access the long-term proxy, so that the WMS can perform the renewal automatically.

By default, the long-term proxy lasts for one week and the proxies created from it last 12 hours. These lifetimes can be changed using the -c and the -t option respectively, but cannot be longer than the lifetime of the user certificate.

You can learn about long term proxies as so:

$ myproxy-info -s <myproxy_server> -d

And you can delete a long-term proxy as so:

$ myproxy-destroy -s <myproxy_server> -d

Steps

Manually delegate your proxy to the (for example) FTS servers by running the following script every 8 hrs via cron:

#!/bin/bash

echo "Refreshing credentials"

# Destroy any existing voms credentials (optional)
# voms-proxy-destroy -debug

# Retrieve a new short term proxy to my UI from RAL the myproxy server with password
myproxy-logon -v -d -s lcgrbp01.gridpp.rl.ac.uk -k renew --stdin_pass < <PATH_TO_DELEGATION_PASSWORD_FILE>

# Stamp the delegated credentials with voms attributes
voms-proxy-init -voms t2k.org:/t2k.org/Role=production -valid 24:0 -noregen

# Delegate the short term voms proxy to the RAL FTS server(s)
glite-delegation-init -f -s https://lcgfts.gridpp.rl.ac.uk:8443/glite-data-transfer-fts/services/FileTransfer -e 840
glite-delegation-init -f -s https://lcgfts3.gridpp.rl.ac.uk:8443/glite-data-transfer-fts/services/gridsite-delegation -e 840 

glite-delegation-init is no longer a supported command - do not use it!!!!