Difference between revisions of "Quick Guide to Dirac"

From GridPP Wiki
Jump to: navigation, search
(Create a proxy for a community)
 
(194 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 +
<p style="color:red">'''This Wiki page has been frozen and will soon become obsolete. The current version can be found under [https://github.com/ic-hep/gridpp-dirac-users/wiki https://github.com/ic-hep/gridpp-dirac-users/wiki].'''</p>
 +
 +
These instruction are for the dirac server at [https://dirac.gridpp.ac.uk/ https://dirac.gridpp.ac.uk/].
 +
 
= Introduction =  
 
= Introduction =  
This document describes how to quickly set up a simple Dirac client, create a job script and then submit it to a Dirac server for execution. You need to have a valid grid certificate, be a member of a supported VO '''and''' you need to be registered with the dirac server (please see below on how to check this) before attempting to install the dirac UI. You '''do not''' need root access to the machine you are installing the Dirac UI on. <br>
+
This document describes how to quickly set up a simple Dirac UI client, create a job script and then submit it to a Dirac server for execution. '''You need to have a valid grid certificate and be a member of a [https://www.gridpp.ac.uk/wiki/VOs_supported_by_gridpp_dirac supported VO].''' You '''do not''' need root access to the machine you are installing the Dirac UI on. <br>
If you are not registered with the dirac server, please contact janusz.martyniak@imperial.ac.uk. <br>
+
By popular request: If you have your grid certificate in a p12 format, you need to convert it to pem files, and place those in the .globus directory in your home directory.
Please consider signing up to the gridpp dirac users mailing list: [https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users Sign up !] <br>
+
<pre>
For operational issues please consider filing a [https://ggus.eu/ GGUS ticket]. If you set "Notify site" to UKI-LT2-IC-HEP, we'll see it quicker. If you don't know if your issue is operational, just use the mailing list :-)
+
openssl pkcs12 -nocerts -in mycert.p12 -out userkey.pem
 +
openssl pkcs12 -clcerts -nokeys -in mycert.p12 -out usercert.pem
 +
chmod 400 userkey.pem (this is a security requirement)
 +
</pre>
 +
''Note: If you are completely new to grid computing, please either talk to computing support in your experiment and/or your local grid admin first to make sure you have the basics covered. Any attempts at producing a "one size fits all" documentation have failed, for good reason. If you have no idea who to contact, please email tb-support/at/jiscmail.ac.uk with as much detail as you are able to provide and we'll work it out from there.''
 +
 
 +
===Getting support===
 +
* Please consider signing up to the gridpp dirac users mailing list: [https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users Sign up !] (It's fairly low traffic.) If you are using a non-institutional email, please let us know, it's hard to seperate the spam from the real requests. Thank you.
 +
* If something isn't working, please try the mailing list first. It may help to include the output from your commands using the debug flag, which can be done with the <code>-ddd</code> flag, e.g.: dirac-proxy-init -ddd -g gridpp_user -M
 +
* For operational issues ("GridPP DIRAC server has died") please consider filing a [https://ggus.eu/ GGUS ticket]. If you set "Notify site" to UKI-LT2-IC-HEP, we'll see it quicker. If you don't know if your issue is operational, just use the mailing list :-)
 +
* If your jobs don't run at a ''specific'' site that you think they should be running at (but run on ''other'' sites), please email lcg-site-admin at imperial.ac.uk. Please include your DN and VO. We will then check if it is a configuration issue on the dirac side and either fix it, or advise you whom to contact instead. Alternatively you can use GGUS for this issue. Please set "Notify site" to UKI-LT2-IC-HEP, we will reroute it to the appropriate site, if applicable.
  
 
= Server URL =  
 
= Server URL =  
The Dirac server is installed at Imperial College. The Web interface is at:
+
The GridPP DIRAC server is installed at Imperial College. The Web interface can be found at:
  
   https://dirac.grid.hep.ph.ic.ac.uk:8443
+
   https://dirac.gridpp.ac.uk
  
 
You need a certificate installed in your browser. <br>
 
You need a certificate installed in your browser. <br>
To check that you are correctly registered with this dirac instance, check the bottom righthand corner. If this shows you as "visitor" and you cannot change this, you are not registered. In this case, please contact Janusz.
+
To check that you are correctly registered with this dirac instance and that you are a member of a dirac supported VO, visit [[https://dirac.gridpp.ac.uk the server]] and check the bottom righthand corner. If this shows you as "visitor" and you cannot change this, you are not registered. In this case, please contact lcg-site-admin@imperial.ac.uk
  
 
= Dirac client installation =  
 
= Dirac client installation =  
 +
'''05/07/2023: Major version upgrade. Please update your UI.''' <br>
 +
If you do not want to install a client and have cvmfs available on your machine you can do:
 
<pre>
 
<pre>
mkdir dirac
+
source /cvmfs/dirac.egi.eu/dirac/bashrc_gridpp
cd dirac
+
dirac-proxy-init -g [your_vo_goes_here]_user -M # (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)
wget -np -O dirac-install http://lhcbproject.web.cern.ch/lhcbproject/dist/Dirac_project/dirac-install
+
chmod +x dirac-install
+
./dirac-install -V gridpp -r v6r11p12
+
source bashrc
+
 
</pre>
 
</pre>
 +
and skip the rest of this paragraph.
  
Note 1: The 'gridpp' in the command above does not refer to the gridpp VO - please use the command as is, independently of which VO you will use later on."
+
''Note: The installation runs in a conda environment and is self-contained, it should work on most Linux platforms. We recommend Rocky or Alma 9''
  
'''Note 2: The version (v6r11p12) should match the version given at the bottom of the [https://dirac00.grid.hep.ph.ic.ac.uk:8443/DIRAC/ GridPP DIRAC portal web pages]. CHECK THIS BEFORE CREATING A NEW DIRAC CLIENT.'''
+
Your user cert and key (usercert.pem/userkey.pem) should be located in the $HOME/.globus directory. You also need to be a member of a GridPP supported VO. <br>
  
= Configure the client =
+
Please check that your firewall is open on the appropriate ports: <br>
 +
nc -v dirac01.grid.hep.ph.ic.ac.uk 9135 <br>
 +
Connection to dirac01.grid.hep.ph.ic.ac.uk 9135 port [tcp/*] succeeded!
  
Create a dumb, groupless proxy (mind -x). This option is to be used only when configuring the client.
+
Currently (July 2023) there is an issue with the latest release of diracos2. Please make sure you install the version indicated in the instructions below, rather that 'latest'.
  
You need your keys (userkey.pem and usercert.pem) in $HOME/.globus/.
+
DIRAC is now python3 only, to install the UI, run this command sequence in a '''bash''' shell:
You will also need the [http://en.wikipedia.org/wiki/Certificate_authority CA] certs. When you install the dirac UI, the script will automatically download a current copy of these certs for you and store them in dirac/etc/grid-security/certificates. This will be sufficient for a couple of weeks of  testing. If you are planning on using your dirac UI for longer, you need to update these certificates regularly. Most grid sites maintain a current copy of these files somewhere on their systems and you should link to those (try /etc/grid-security/certificates or at Imperial /vols/grid/certificates) instead.
+
  
Make a proxy.
 
 
<pre>
 
<pre>
$ dirac-proxy-init -x
+
mkdir dirac_ui
 +
cd dirac_ui
 +
curl -LO https://github.com/DIRACGrid/DIRACOS2/releases/download/2.38/DIRACOS-Linux-x86_64.sh
 +
bash DIRACOS-Linux-$(uname -m).sh
 +
rm DIRACOS-Linux-$(uname -m).sh
 +
source diracos/diracosrc
 +
pip install DIRAC==8.0.38
 +
dirac-proxy-init -x -N
 +
(ignore message about "No CRL files found" - these will be installed with the next step)
 +
dirac-configure -F -S GridPP -C dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server -I
  
Generating proxy...
+
dirac-proxy-init -g [your_vo_goes_here]_user -M # (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)
Enter Certificate password:
+
Proxy generated:
+
subject      : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=janusz martyniak/CN=proxy
+
issuer      : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=janusz martyniak
+
identity    : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=janusz martyniak
+
timeleft    : 23:59:59
+
path        : /tmp/x509up_u500
+
username    : unknown
+
properties  : 
+
 
</pre>
 
</pre>
  
 +
If you see an error like below then you probably are not registered with the dirac instance. Usually you are automatically registered, but there is a delay of several hours between registering with the VO and the dirac server picking this up. If after 12h you still get the same error message, please send an email to lcg-site-admin -at- imperial.ac.uk and we'll have a look.
  
Configure defaults. (Also use this command (-F == force update) when updating the UI).
 
 
<pre>
 
$ dirac-configure -F defaults-gridpp.cfg
 
</pre>
 
 
if you see an error like below then you probably are not registered with the dirac instance.
 
 
<pre>
 
<pre>
 
Could not sync dir Cannot get URL for Framework/BundleDelivery in setup MyDIRAC-Production: Option /DIRAC/Setups/MyDIRAC-Production/Framework is not defined
 
Could not sync dir Cannot get URL for Framework/BundleDelivery in setup MyDIRAC-Production: Option /DIRAC/Setups/MyDIRAC-Production/Framework is not defined
 
</pre>
 
</pre>
  
= Create a proxy for a community=  
+
= Submitting a 'Hello World' job via dirac =
  
To be able to submit a job, you need to be a member of a community (VO), here '''gridpp'''. If you are, you can create a proxy:
+
== Create a proxy ==
  
<pre>
+
If you don't already have a valid proxy you should create one:
$ dirac-proxy-init -g gridpp_user -M
+
Generating proxy...
+
Enter Certificate password:
+
Uploading proxy for gridpp_user...
+
  
Proxy generated:
+
<pre>
subject      : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=janusz martyniak/CN=proxy
+
source diracos/diracosrc
issuer      : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=janusz martyniak
+
dirac-proxy-init -g [your_vo_goes_here]_user (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)
identity    : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=janusz martyniak
+
timeleft    : 23:59:58
+
DIRAC group  : gridpp_user
+
path        : /tmp/x509up_u500
+
username    : martynia
+
properties  : NormalUser
+
 
</pre>
 
</pre>
  
You can easily verify what VOs you are a member of, by going to the Dirac website and clicking at the drop-down menu at the bottom right, next to your username@ text. If your username shows "Anonymous" it means
+
== Create a JDL ==
that you are not registered as a Dirac user. Please contact janusz.martyniak at imperial.ac.uk. <br>
+
'''Other VOs: vo.londongrid.ac.uk: use londongrid_user, for t2k.org please use t2k_user, for pheno use pheno_user, for snoplus use snoplus_user, for cernatschool use cernatschool_user, comet: comet_user''' <br>
+
Note: We are aware that the VO name and the dirac group should match, but for the time being, please bear with us and use the dirac groups ......
+
 
+
= Create a JDL =
+
  
 
Now we are ready to create a simple Dirac jdl. Put the lines below into a file Simple.jdl:
 
Now we are ready to create a simple Dirac jdl. Put the lines below into a file Simple.jdl:
Line 103: Line 95:
 
</pre>
 
</pre>
  
= Submit a job =
+
== Submit a job ==
 
+
  
 
<pre>
 
<pre>
$ dirac-wms-job-submit Simple.jdl  
+
$ dirac-wms-job-submit -f logfile Simple.jdl  
 
JobID = 236
 
JobID = 236
 
</pre>
 
</pre>
Line 114: Line 105:
 
   
 
   
 
<pre>
 
<pre>
$ dirac-wms-job-status 236
+
$ dirac-wms-job-status -f logfile
  
 
JobID=236 Status=Waiting; MinorStatus=Pilot Agent Submission; Site=ANY  
 
JobID=236 Status=Waiting; MinorStatus=Pilot Agent Submission; Site=ANY  
Line 122: Line 113:
  
 
<pre>
 
<pre>
$ dirac-wms-job-status 236
+
$ dirac-wms-job-status -f logfile
 
JobID=236 Status=Done; MinorStatus=Execution Complete; Site=LCG.Glasgow.uk;
 
JobID=236 Status=Done; MinorStatus=Execution Complete; Site=LCG.Glasgow.uk;
 
</pre>
 
</pre>
Line 129: Line 120:
 
You can also submit a job using the Web interface (Tools->Job Launchpad).
 
You can also submit a job using the Web interface (Tools->Job Launchpad).
  
 +
Once the job is done, retrieve the output:
 +
<pre>
 +
$ dirac-wms-job-get-output -f logfile
 +
</pre>
 +
 +
A list of JDL parameters used by DIRAC can be found in the [https://dirac.readthedocs.io/en/latest/UserGuide/GettingStarted/UserJobs/JDLReference/index.html DIRAC documentation].
 +
 +
= DIRAC Job Execution Environment =
 +
 +
Jobs running within DIRAC will start with an environment similar to the python3 DIRAC UI; this means that by default:
 +
* All dirac-* commands (such as dirac-dms-add-file) are available within job scripts.
 +
* Running "python" will start python3 with the DIRAC API ("import DIRAC") module available.
 +
* The system version of python (python2) is available if called by "python2" or the full path: /usr/bin/python.
 +
* Other grid tools, such as gfal_utils (gfal-copy, etc.) are available in a configured and working state.
 +
* A few other commands (curl, openssl, perl) will use the DIRAC versions by default. You may need to call the system versions with the full path if you specifically need the system version for any reason.
 +
 +
It is recommended that you run one of your jobs locally while you have the DIRAC UI sourced and check that everything works before submitting to the grid. As always, if you find any problems you need help with, please contact the gridpp-dirac-users list.
 +
 +
== Classic Grid WN Environment ==
 +
 +
There may be some cases where you want to specifically run the classic grid WN toolset: python2 + gfal_utils without the DIRAC tools. This can be achieved by resetting the PATH and activating the software from CVMFS in your job script:
 +
 +
<pre>
 +
#!/bin/bash
 +
export PATH="/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin"
 +
source /cvmfs/grid.cern.ch/umd-c7wn-latest/etc/profile.d/setup-c7-wn-example.sh
 +
env # For debugging
 +
[the rest of your job here]
 +
</pre>
 +
 +
This can be used in a sub-shell rather than at the start of the job if you want to mix between the classic and DIRAC environments, but this is probably best avoided unless absolutely necessary.
 +
 +
= Advanced job management =
 +
For any real work, we recommend that you develop your code against the DIRAC python API. Please also read the "Data Handling Within a Job" section before you produce any kind of data to be stored on a storage element.
 +
== Using the DIRAC API to submit a Job ==
 +
 +
An example of a hello world job can be found
 +
[https://github.com/ic-hep/DIRAC-tools/blob/master/user/hello_world_job.py here]. <br>
 +
Another example with more options: [https://github.com/ic-hep/DIRAC-tools/blob/master/user/job_with_more_options.py Options!]. You will also need
 +
[https://github.com/ic-hep/DIRAC-tools/blob/master/user/testapi.sh this] shell script which is used as an executable within the example. <br>
 +
The corresponding section in the DIRAC user guide can be found [http://dirac.readthedocs.io/en/latest/UserGuide/GettingStarted/UserJobs/DiracAPI/index.html here] and the Job API [https://dirac.readthedocs.io/en/latest/CodeDocumentation/Interfaces/API/Job.html here].
 +
 +
== Selecting a site platform ==
 +
 +
By default GridPP DIRAC will only run jobs at sites providing an EL7 (as in "Enterprise Linux 7", treat it as a synonym for SL7) based operating system.
 +
If you would like to submit a job that requires a different operating system (such as EL8 or EL9)
 +
this can be set using the Platform option in the JDL:
 +
 +
<pre>Platform = "EL9";</pre>
 +
 +
If is also possible to specify AnyPlatform as the platform name to disable this check:
 +
 +
<pre>Platform = "AnyPlatform";</pre>
 +
 +
Note that all parts of this JDL string are case sensitive (the word Platform and the platform string itself).
 +
 +
Users of the DIRAC API can use the '''job.setPlatform("EL8")''' function to configure this behaviour.
 +
 +
Ganga users can set extra options, including the platform, via the backend settings attribute:
 +
 +
<pre>
 +
j = Job(backend=Dirac())
 +
j.backend.settings['Platform'] = "EL9"
 +
</pre>
 +
 +
== Advanced Site/CE selection ==
 +
 +
If you want to run at a specific subset of available sites for your VO, you can specify a list of Sites
 +
using the usual Site clause with a list:
 +
 +
<pre>
 +
Site = {"Site1", "Site2", "Site3"};
 +
</pre>
 +
 +
It is also possible to select specific CEs (generally this is used for targeting special resources):
 +
 +
<pre>
 +
GridCE = {"ceprod05.grid.hep.ph.ic.ac.uk", "ceprod06.grid.hep.ph.ic.ac.uk"};
 +
</pre>
 +
 +
Note that selecting an invalid combination of Sites & CEs may result in your job staying in the waiting state forever.
 +
 +
= Basic data management =
 +
 +
This is for use with the UI and proxy you have just set up. For data handling within a job, please see: [[DIRAC_Data_Handling_within_a_Job]] <br>
 +
<br>
 +
 +
List files in the dirac file catalogue (please do not use the file catalogue for anything else):
 +
<pre>
 +
$ dirac-dms-filecatalog-cli
 +
Starting FileCatalog client
 +
File Catalog Client $Revision: 1.17 $Date:
 +
FC:/> ls
 +
(use 'exit' to, well, exit)
 +
</pre>
 +
 +
Add a file to an SE (here UKI-LT2-IC-HEP-disk). Please replace /gridpp with the name of the VO you are using, and firstname.lastname with your first and last name (or alias or superhero, it just has to be unique):
 +
<pre>
 +
dirac-dms-add-file /gridpp/user/f/firstname.lastname/myfile.txt myfile.txt UKI-LT2-IC-HEP-disk
 +
</pre>
 +
 +
List all replicas:
 +
<pre>
 +
dirac-dms-lfn-replicas /gridpp/user/f/firstname.lastname/myfile.txt
 +
</pre>
 +
 +
Copy file from SE to disk: <br>
 +
<pre>
 +
dirac-dms-get-file /gridpp/user/f/firstname.lastname/myfile.txt
 +
</pre>
 +
 +
Replicate file from on SE to another (file is currently at UKI-LT2-IC-HEP-disk): <br>
 +
<pre>
 +
dirac-dms-replicate-lfn /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-QMUL2-disk
 +
</pre>
 +
 +
Ask for the access URL (you should only need this for very special cases):
 +
<pre>
 +
dirac-dms-lfn-accessURL  /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-IC-HEP-disk
 +
</pre>
 +
 +
Remove file from a specific SE: <br>
 +
<pre>
 +
dirac-dms-remove-replicas /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-IC-HEP-disk
 +
</pre>
 +
 +
Remove all replicas (?):
 +
<pre>
 +
dirac-dms-remove-files  /gridpp/user/f/firstname.lastname/myfile.txt
 +
</pre>
 +
 +
<hr>
 +
Using the asynchronous files transfer tools.  <br>
 +
Replicate a file to a given SE:
 +
<pre>
 +
dirac-dms-replicate-and-register-request [make up an id for this transfer] [LFN] [target SE]
 +
</pre>
 +
e.g. replicate a file to the RHUL SE (files must exist and be registered in the file catalogue):
 +
<pre>
 +
dirac-dms-replicate-and-register-request daniela1 /gridpp/user/d/daniela.bauer/small.text.file.txt UKI-LT2-RHUL-disk
 +
</pre>
 +
Querying the transfers:
 +
<pre>
 +
dirac-rms-request [transfer if from above, e.g. daniela1]
 +
dirac-rms-request --Status='Done' --Since='2016-05-05'
 +
</pre>
 +
Resetting a transfer (this will only resubmit the failed transfers:
 +
<pre>
 +
dirac-rms-request --Reset [transferid]
 +
</pre>
 +
 +
=== Metadata ===
 +
The DIRAC File Catalog allows you to associate metadata with your files. You can find a basic introduction here:
 +
[[DIRAC_FileCatalog_MetaData]]
 +
<br>Note that currently metadata is not heavily used, please report any bugs to us, so we can try to fix them.
 +
 +
=Input and Output data handling within a job=
 +
 +
Please see the documentation here: [[DIRAC_Data_Handling_within_a_Job]]
 +
 +
=Advanced Topics=
 +
 +
== Experimental GPU Support ==
 +
 +
For details of how to use GPUs with DIRAC see [[ GPU Support ]]
  
 +
=DIRAC command reference=
 +
[https://dirac.readthedocs.io/en/latest/ DIRAC on ReadTheDocs]
  
 +
<br />
 
<hr>
 
<hr>
 
Back to [https://www.gridpp.ac.uk/wiki/Dirac GridPP Dirac] overview page.
 
Back to [https://www.gridpp.ac.uk/wiki/Dirac GridPP Dirac] overview page.

Latest revision as of 12:21, 25 June 2024

This Wiki page has been frozen and will soon become obsolete. The current version can be found under https://github.com/ic-hep/gridpp-dirac-users/wiki.

These instruction are for the dirac server at https://dirac.gridpp.ac.uk/.

Introduction

This document describes how to quickly set up a simple Dirac UI client, create a job script and then submit it to a Dirac server for execution. You need to have a valid grid certificate and be a member of a supported VO. You do not need root access to the machine you are installing the Dirac UI on.
By popular request: If you have your grid certificate in a p12 format, you need to convert it to pem files, and place those in the .globus directory in your home directory.

openssl pkcs12 -nocerts -in mycert.p12 -out userkey.pem
openssl pkcs12 -clcerts -nokeys -in mycert.p12 -out usercert.pem
chmod 400 userkey.pem (this is a security requirement)

Note: If you are completely new to grid computing, please either talk to computing support in your experiment and/or your local grid admin first to make sure you have the basics covered. Any attempts at producing a "one size fits all" documentation have failed, for good reason. If you have no idea who to contact, please email tb-support/at/jiscmail.ac.uk with as much detail as you are able to provide and we'll work it out from there.

Getting support

  • Please consider signing up to the gridpp dirac users mailing list: Sign up ! (It's fairly low traffic.) If you are using a non-institutional email, please let us know, it's hard to seperate the spam from the real requests. Thank you.
  • If something isn't working, please try the mailing list first. It may help to include the output from your commands using the debug flag, which can be done with the -ddd flag, e.g.: dirac-proxy-init -ddd -g gridpp_user -M
  • For operational issues ("GridPP DIRAC server has died") please consider filing a GGUS ticket. If you set "Notify site" to UKI-LT2-IC-HEP, we'll see it quicker. If you don't know if your issue is operational, just use the mailing list :-)
  • If your jobs don't run at a specific site that you think they should be running at (but run on other sites), please email lcg-site-admin at imperial.ac.uk. Please include your DN and VO. We will then check if it is a configuration issue on the dirac side and either fix it, or advise you whom to contact instead. Alternatively you can use GGUS for this issue. Please set "Notify site" to UKI-LT2-IC-HEP, we will reroute it to the appropriate site, if applicable.

Server URL

The GridPP DIRAC server is installed at Imperial College. The Web interface can be found at:

 https://dirac.gridpp.ac.uk

You need a certificate installed in your browser.
To check that you are correctly registered with this dirac instance and that you are a member of a dirac supported VO, visit [the server] and check the bottom righthand corner. If this shows you as "visitor" and you cannot change this, you are not registered. In this case, please contact lcg-site-admin@imperial.ac.uk

Dirac client installation

05/07/2023: Major version upgrade. Please update your UI.
If you do not want to install a client and have cvmfs available on your machine you can do:

source /cvmfs/dirac.egi.eu/dirac/bashrc_gridpp
dirac-proxy-init -g [your_vo_goes_here]_user -M # (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)

and skip the rest of this paragraph.

Note: The installation runs in a conda environment and is self-contained, it should work on most Linux platforms. We recommend Rocky or Alma 9

Your user cert and key (usercert.pem/userkey.pem) should be located in the $HOME/.globus directory. You also need to be a member of a GridPP supported VO.

Please check that your firewall is open on the appropriate ports:
nc -v dirac01.grid.hep.ph.ic.ac.uk 9135
Connection to dirac01.grid.hep.ph.ic.ac.uk 9135 port [tcp/*] succeeded!

Currently (July 2023) there is an issue with the latest release of diracos2. Please make sure you install the version indicated in the instructions below, rather that 'latest'.

DIRAC is now python3 only, to install the UI, run this command sequence in a bash shell:

mkdir dirac_ui
cd dirac_ui
curl -LO https://github.com/DIRACGrid/DIRACOS2/releases/download/2.38/DIRACOS-Linux-x86_64.sh
bash DIRACOS-Linux-$(uname -m).sh
rm DIRACOS-Linux-$(uname -m).sh
source diracos/diracosrc
pip install DIRAC==8.0.38
dirac-proxy-init -x -N
(ignore message about "No CRL files found" - these will be installed with the next step)
dirac-configure -F -S GridPP -C dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server -I

dirac-proxy-init -g [your_vo_goes_here]_user -M # (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)

If you see an error like below then you probably are not registered with the dirac instance. Usually you are automatically registered, but there is a delay of several hours between registering with the VO and the dirac server picking this up. If after 12h you still get the same error message, please send an email to lcg-site-admin -at- imperial.ac.uk and we'll have a look.

Could not sync dir Cannot get URL for Framework/BundleDelivery in setup MyDIRAC-Production: Option /DIRAC/Setups/MyDIRAC-Production/Framework is not defined

Submitting a 'Hello World' job via dirac

Create a proxy

If you don't already have a valid proxy you should create one:

source diracos/diracosrc
dirac-proxy-init -g [your_vo_goes_here]_user (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)

Create a JDL

Now we are ready to create a simple Dirac jdl. Put the lines below into a file Simple.jdl:

[
JobName = "Simple_Job";
Executable = "/bin/ls";
Arguments = "-ltr";
StdOutput = "StdOut";
StdError = "StdErr";
OutputSandbox = {"StdOut","StdErr"};
]

Submit a job

$ dirac-wms-job-submit -f logfile Simple.jdl 
JobID = 236

Check job status:

$ dirac-wms-job-status -f logfile

JobID=236 Status=Waiting; MinorStatus=Pilot Agent Submission; Site=ANY 

and eventually:

$ dirac-wms-job-status -f logfile 
JobID=236 Status=Done; MinorStatus=Execution Complete; Site=LCG.Glasgow.uk;

The jobs can also be monitored using the Web interface. Go to Jobs ->Job monitor You can also submit a job using the Web interface (Tools->Job Launchpad).

Once the job is done, retrieve the output:

$ dirac-wms-job-get-output -f logfile

A list of JDL parameters used by DIRAC can be found in the DIRAC documentation.

DIRAC Job Execution Environment

Jobs running within DIRAC will start with an environment similar to the python3 DIRAC UI; this means that by default:

  • All dirac-* commands (such as dirac-dms-add-file) are available within job scripts.
  • Running "python" will start python3 with the DIRAC API ("import DIRAC") module available.
  • The system version of python (python2) is available if called by "python2" or the full path: /usr/bin/python.
  • Other grid tools, such as gfal_utils (gfal-copy, etc.) are available in a configured and working state.
  • A few other commands (curl, openssl, perl) will use the DIRAC versions by default. You may need to call the system versions with the full path if you specifically need the system version for any reason.

It is recommended that you run one of your jobs locally while you have the DIRAC UI sourced and check that everything works before submitting to the grid. As always, if you find any problems you need help with, please contact the gridpp-dirac-users list.

Classic Grid WN Environment

There may be some cases where you want to specifically run the classic grid WN toolset: python2 + gfal_utils without the DIRAC tools. This can be achieved by resetting the PATH and activating the software from CVMFS in your job script:

#!/bin/bash
export PATH="/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin"
source /cvmfs/grid.cern.ch/umd-c7wn-latest/etc/profile.d/setup-c7-wn-example.sh
env # For debugging
[the rest of your job here]

This can be used in a sub-shell rather than at the start of the job if you want to mix between the classic and DIRAC environments, but this is probably best avoided unless absolutely necessary.

Advanced job management

For any real work, we recommend that you develop your code against the DIRAC python API. Please also read the "Data Handling Within a Job" section before you produce any kind of data to be stored on a storage element.

Using the DIRAC API to submit a Job

An example of a hello world job can be found here.
Another example with more options: Options!. You will also need this shell script which is used as an executable within the example.
The corresponding section in the DIRAC user guide can be found here and the Job API here.

Selecting a site platform

By default GridPP DIRAC will only run jobs at sites providing an EL7 (as in "Enterprise Linux 7", treat it as a synonym for SL7) based operating system. If you would like to submit a job that requires a different operating system (such as EL8 or EL9) this can be set using the Platform option in the JDL:

Platform = "EL9";

If is also possible to specify AnyPlatform as the platform name to disable this check:

Platform = "AnyPlatform";

Note that all parts of this JDL string are case sensitive (the word Platform and the platform string itself).

Users of the DIRAC API can use the job.setPlatform("EL8") function to configure this behaviour.

Ganga users can set extra options, including the platform, via the backend settings attribute:

j = Job(backend=Dirac())
j.backend.settings['Platform'] = "EL9"

Advanced Site/CE selection

If you want to run at a specific subset of available sites for your VO, you can specify a list of Sites using the usual Site clause with a list:

Site = {"Site1", "Site2", "Site3"};

It is also possible to select specific CEs (generally this is used for targeting special resources):

GridCE = {"ceprod05.grid.hep.ph.ic.ac.uk", "ceprod06.grid.hep.ph.ic.ac.uk"};

Note that selecting an invalid combination of Sites & CEs may result in your job staying in the waiting state forever.

Basic data management

This is for use with the UI and proxy you have just set up. For data handling within a job, please see: DIRAC_Data_Handling_within_a_Job

List files in the dirac file catalogue (please do not use the file catalogue for anything else):

$ dirac-dms-filecatalog-cli
Starting FileCatalog client
File Catalog Client $Revision: 1.17 $Date: 
FC:/> ls
(use 'exit' to, well, exit)

Add a file to an SE (here UKI-LT2-IC-HEP-disk). Please replace /gridpp with the name of the VO you are using, and firstname.lastname with your first and last name (or alias or superhero, it just has to be unique):

dirac-dms-add-file /gridpp/user/f/firstname.lastname/myfile.txt myfile.txt UKI-LT2-IC-HEP-disk

List all replicas:

dirac-dms-lfn-replicas /gridpp/user/f/firstname.lastname/myfile.txt

Copy file from SE to disk:

dirac-dms-get-file /gridpp/user/f/firstname.lastname/myfile.txt

Replicate file from on SE to another (file is currently at UKI-LT2-IC-HEP-disk):

dirac-dms-replicate-lfn /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-QMUL2-disk

Ask for the access URL (you should only need this for very special cases):

dirac-dms-lfn-accessURL  /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-IC-HEP-disk

Remove file from a specific SE:

dirac-dms-remove-replicas /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-IC-HEP-disk

Remove all replicas (?):

dirac-dms-remove-files  /gridpp/user/f/firstname.lastname/myfile.txt

Using the asynchronous files transfer tools.
Replicate a file to a given SE:

dirac-dms-replicate-and-register-request [make up an id for this transfer] [LFN] [target SE]

e.g. replicate a file to the RHUL SE (files must exist and be registered in the file catalogue):

dirac-dms-replicate-and-register-request daniela1 /gridpp/user/d/daniela.bauer/small.text.file.txt UKI-LT2-RHUL-disk

Querying the transfers:

dirac-rms-request [transfer if from above, e.g. daniela1]
dirac-rms-request --Status='Done' --Since='2016-05-05'

Resetting a transfer (this will only resubmit the failed transfers:

dirac-rms-request --Reset [transferid]

Metadata

The DIRAC File Catalog allows you to associate metadata with your files. You can find a basic introduction here: DIRAC_FileCatalog_MetaData
Note that currently metadata is not heavily used, please report any bugs to us, so we can try to fix them.

Input and Output data handling within a job

Please see the documentation here: DIRAC_Data_Handling_within_a_Job

Advanced Topics

Experimental GPU Support

For details of how to use GPUs with DIRAC see GPU Support

DIRAC command reference

DIRAC on ReadTheDocs



Back to GridPP Dirac overview page.