Local File Access TestSuite
Introduction
A (very) simple series of scripts that I use for submitting jobs to test local file access. They simply provide a series of bash wrapper scripts around glite-wms-job submission tools, or batch system commands, to run a series of tests described below. The scripts are minimal so should be readable to determine what they do. They have not been user-tested so may have bugs. I am happy to add features to make them useful to others - but there is no point in reinventing ganga or hammercloud.
Download
A currently very-beta version of the scripts is available at: SRMLocalAccesTestSuite-0.1
Usage
- Define the type of test and parameters in a file like those in the parameters directory.
- sumbit jobs with ./submitJobs.sh [parameters/yourfile]
- ./checkJobs.sh [parameters/yourfile] - when all say done will say Done
- ./retrieveOutput.sh when all are done
- Look in Outputs for results including stdout/err and a stats file.
You may want to change the VirtualOrganisation in the jdltemplates/ if you are not in dteam or atlas VOs.
Local batch system version
You should be able to submit the scripts in testscripts on the batch system locally. You will need to make sure that somehow the following is done in the script or before it runs:
source $GLITE_LOCATION/external/etc/profile.d/grid-env.sh export X509_USER_PROXY=~/x509up_u52139
where the later points at an existing proxy (made with, e.g. voms-proxy-init).
Todo/ Requests
- Provide source files and options for IoPerformer
- Complete processOutput so that it displays a plot of cpu time and something.
- Another thing that it doesn't do that would be (quite) easy to add is to automatically resubmit like hammercloud. I am happy to give that a go if people want but logic would have to be right so it didn't go ballistic and again I don't want to put in a lot of work to recreate the hammercloud.
README
A very simple script to sumbit jobs via the wms for use in testing storage.
Instructions for use
parameters.sh
Specify parameters for the test in here. Or make your own copy of this and pass as an argument to submitAthJobs.sh
Variables are TESTTYPE [=athena,ioperformer,rfcp] athena runs a D3PDMaker job. ioperfomer - opens the files over rfio and reads through them in a simple root app that returns IO info on the main TTree rfcp - simply copies the file to the WN - and deletes it.
RELEASE [=15.6.9] select a atlas sw release available at the site in question (see for example http://pprc.qmul.ac.uk/~lloyd/gridpp/atest.html for the latest available at each site)
CE [=mw05.ecdf.ed.ac.uk:2119/jobmanager-lcgsge-ecdf] specify both node and queue name. This can also be got from one of the jobs submitted at http://pprc.qmul.ac.uk/~lloyd/gridpp/atest.html for the latest available at each sit if you don't know it
INFILE [infiles.txt] A file (in the Inputs directory) that contains a list of files to run on (each as a seperate line - at least two of them and a CR at the end) For DPMs reading files over rfio: rfio:/dpm/ecdf.ed.ac.uk/home/atlas/atlasscratchdisk/AOD.065320._00159.pool.root.3.4504984.0
(If you want to run on files from a particular server then to generate the list of surls run something like dpm-disk-to-dpns -sSERVER:FS | cut -d " " -f2 | sed 's%^/dpm%rfio:/dpm%'
For Storm or anything where physical filenames can be used: /mnt/lustre/atlas/atlasscratchdisk/AOD.065320._00159.pool.root.3.4504984.0
2) Specify test type, release, input file name and CE/queue in parameters.sh
submitAthJobs.sh [parameters file]
sumbits the jobs - optional argument of parameters file (otherwise assumes parameters.sh)
checkTestJobs.sh
checks the jobs - simply reports the status
retrieveOutput.sh
gets the ouputs of "Done" jobs - puts them in a directory.
processOutput.sh
processes the output directory to produce a space seperated file of real system user times. A root exe to make charts from these will be made available.
