Difference between revisions of "DIRAC Data Handling within a Job"

From GridPP Wiki
Jump to: navigation, search
(now with API syntax as well)
Line 1: Line 1:
==Using JDLs==
 
 
'''Note: Please ensure that you always specify the storage element you want your data to be written to.''' This must be a storage element you have write access to. We strongly recommend to upload a file to your chosen storage element by hand before you submit a job to check that the permissions on the storage element are set correctly.  
 
'''Note: Please ensure that you always specify the storage element you want your data to be written to.''' This must be a storage element you have write access to. We strongly recommend to upload a file to your chosen storage element by hand before you submit a job to check that the permissions on the storage element are set correctly.  
 +
 +
==Using JDLs==
 +
 
* Specify OutputSE and OutputData <br /> Example: Your job produces files ending in txt and dat and you want all of these files to go to the storage element at Imperial:
 
* Specify OutputSE and OutputData <br /> Example: Your job produces files ending in txt and dat and you want all of these files to go to the storage element at Imperial:
 
<pre>
 
<pre>

Revision as of 15:54, 12 June 2020

Note: Please ensure that you always specify the storage element you want your data to be written to. This must be a storage element you have write access to. We strongly recommend to upload a file to your chosen storage element by hand before you submit a job to check that the permissions on the storage element are set correctly.

Using JDLs

  • Specify OutputSE and OutputData
    Example: Your job produces files ending in txt and dat and you want all of these files to go to the storage element at Imperial:
OutputSE = "UKI-LT2-IC-HEP-disk";
OutputData = { "*.txt", "*.dat" }; 

By default these files are uploaded to the user area (/voname/user/initial/your.dirac.username/), into a directory that references the job number (here '700'):
In my example this is: /gridpp/user/d/daniela.bauer/0/700/

FC:/> ls /gridpp/user/d/daniela.bauer/0/700/
testfile.1591886794.2.LCG.UKI-LT2-IC-HEP.uk.txt
testfile.1591886794.LCG.UKI-LT2-IC-HEP.uk.dat
testfile.1591886794.LCG.UKI-LT2-IC-HEP.uk.txt
  • Specify OutputSE, OutputData and OutputPath (recommended)
    Example:
OutputSE = "UKI-LT2-IC-HEP-disk";
OutputPath = "/special_path";
OutputData = { "*.txt", "*.dat" }; 

Now the files will still be in the user area, but located in the directory you specified:

FC:/> ls /gridpp/user/d/daniela.bauer/special_path/
testfile.1591886807.2.LCG.UKI-LT2-IC-HEP.uk.txt
testfile.1591886807.LCG.UKI-LT2-IC-HEP.uk.dat
testfile.1591886807.LCG.UKI-LT2-IC-HEP.uk.txt
  • Specify full LFN (no wildcards possible)
    Example: Your job produces three outputfiles. The first two need to go in one directory, the last one into a different one.
  • Do everything by hand inside your job. In this case you do not need to specify anything in your JDL, put please make sure you include at least one retry for your file upload, to deal with intermittent problems:
    Example (bash):
dirac-dms-add-file /gridpp/daniela.bauer/outputexamples/txt/testfile1.txt testfile1.txt UKI-LT2-IC-HEP-disk
    # if you do this, please always check if your upload has worked, and of not, try again in a couple of minutes, as failures are often transient
    echo $?
    if [ $? -ne 0 ]; then
        # wait 5 min and try again
        sleep 300
        dirac-dms-add-file /gridpp/daniela.bauer/outputexamples/txt/testfile1.txt testfile1.txt UKI-LT2-IC-HEP-disk
        echo $?
        if [ $? -ne 0 ]; then
            echo "Upload failed even in second try."
        fi
    fi

Using the Python API

This assumes you have a job object (do they call it that in python?):

from DIRAC.Interfaces.API.Job import Job
job = Job()
  • Specify output storage element and output data only (wildcards possible). As above, output will appear in /voname/user/initial/your.dirac.username/[]/jobnumber.
job.setOutputData(['*.txt', '*.dat'], outputSE='UKI-LT2-IC-HEP-disk')
  • Specify output storage element, directory and output data.
job.setOutputData(['*.txt', '*.dat'], outputSE='UKI-LT2-IC-HEP-disk', outputPath='/api/specialpath')