Long running jobs using myproxy
*** THIS PAGE CONTAINS OBSOLETE INFORMATION ***
This is a solution to a well-known problem whereby long glite file transfers or other long operations fail if they outlive the lifetime of the voms proxy at time of submission.
Introduction and context
A decent explanation of the workings of long term proxies is given here: http://www.gridpp.ac.uk/deployment/users/myproxy.html
The user guide also contains much contextual information on proxy delegation: https://edms.cern.ch/file/722398/1.4/gLite-3-UserGuide.pdf
Please read and understand those sources before trying this procedure.
Note that there must be a valid proxy on the UI, created with grid-proxy-init or voms-proxy-init, to successfully interact with a long-term proxy on the MyProxy server. So the first step is to create a voms proxy for your vo:
$ voms-proxy-init -voms <vo>
You can query your proxy as follows:
$ voms-proxy-info -all
And it can be destroyed as so
Proxy Renewal Concepts
Proxies created as described in the previous section pose a problem: if a job does not finish before the expiration time of the proxy used to submit it, is aborted. This can easily happen, for example, if the job takes a very long time to execute, or if it stays in a queue for a long time. The easiest solution to the problem would be to use very long-lived proxies, but at the expense of an increased security risk. To overcome this limitation, a proxy credential repository system is used, which allows the user to create and store a long-term proxy in a dedicated server (a MyProxy server). The WMS will then be able to use this long-term proxy to periodically renew the proxy for a submitted job before it expires and until the job ends (or the long-term proxy expires). A UI may have a default server defined in the MYPROXY SERVER environment variable. Management of the proxy renewal functionality is available via the myproxy commands.
Long-term proxy lifecycle
To create and store a long-term proxy:
$ myproxy-init -s <myproxy_server> -d -n
where -s <myproxy server> specifies the hostname of the machine where a MyProxy Server runs, the -d option instructs the server to associate the user DN to the proxy, and the -n option avoids the use of a passphrase to access the long-term proxy, so that the WMS can perform the renewal automatically.
By default, the long-term proxy lasts for one week and the proxies created from it last 12 hours. These lifetimes can be changed using the -c and the -t option respectively, but cannot be longer than the lifetime of the user certificate.
You can learn about long term proxies as so:
$ myproxy-info -s <myproxy_server> -d
And you can delete a long-term proxy as so:
$ myproxy-destroy -s <myproxy_server> -d
Manually delegate your proxy to the (for example) FTS servers by running the following script every 8 hrs via cron:
#!/bin/bash echo "Refreshing credentials" # Destroy any existing voms credentials (optional) # voms-proxy-destroy -debug # Retrieve a new short term proxy to my UI from RAL the myproxy server with password myproxy-logon -v -d -s lcgrbp01.gridpp.rl.ac.uk -k renew --stdin_pass < <PATH_TO_DELEGATION_PASSWORD_FILE> # Stamp the delegated credentials with voms attributes voms-proxy-init -voms t2k.org:/t2k.org/Role=production -valid 24:0 -noregen # Delegate the short term voms proxy to the RAL FTS server(s) glite-delegation-init -f -s https://lcgfts.gridpp.rl.ac.uk:8443/glite-data-transfer-fts/services/FileTransfer -e 840 glite-delegation-init -f -s https://lcgfts3.gridpp.rl.ac.uk:8443/glite-data-transfer-fts/services/gridsite-delegation -e 840
glite-delegation-init is no longer a supported command - do not use it!!!!