Difference between revisions of "BaBar Skimming Creating a New Tarball"
Chris brew (Talk | contribs) |
(No difference)
|
Latest revision as of 10:10, 14 March 2008
This ios an outline of the procedure used to create a new tarball for installation on the grid from a basic Release + Tags.
Contents
Set up your Base Release
Set up the new release and tags as normal
newrel -t 24.2.1d Test-24.2.1d cd Test-24.2.1d srtpath gmake installdirs addpkg BbkJobWrappers cajb20080225a addpkg BbkTaskManager tja080225a addpkg BbkTMGridTools cajb20080225a addpkg BbkSqlAbstractor tja071123a addpkg BbkTools tja071123a addpkg workdir workdir V00-04-21 gmake workdir.setup gmake binscripts
Next create a test task in the normal way and make sure you can run skims locally.
Create and run an strace Task
In BbkTaskManager/site/RAL
there is a task template file RAL-strace-Task-Template
. You need to create a task based on that template + whatever is needed to get your release to run over it's validation dataset. You need to keep
SkimShellTemplateFile = runTemplate-strace
and make sure that the SkimMaxEvents
is set high enough that only one skim event per input collection is created, runTemplate-strace will mangle the tcl
file to only run on the first ten events of the input file.
Now create and submit the skim jobs for the strace task, the should be submitted to the express queue and run pretty quickly creating a .strace
file in each skim run directory.
Create the file list and the Grid Release
Create a file with the list of .strace
files, something like:
find /stage/xrootd-data6/bbrprod/skimming/tasks/RAL-strace-24.2.1d-1.2/28/ -name '*.strace' > trace-files.lis
Then use the mk-lists.sh
script to process the strace output and produce a list of files:
./mk-lists.sh trace-files.lis
Next edit the mk-tarball.sh
script to change the output directory and the versions of BbkTaskmanager, BbkJobWrappers, BbkTools and BbkSqlAbstractor it will add to the grid release, and run the script on the output of the mk-lists.sh
script:
./mk-tarball.pl files.lis
Testing the Grid Release
Create a new Test Task that would run on the grid at the RAL using the RAL xrootd for data input and output.
The create a test run with:
BbkSubmitSkims --test -n 1 <TaskName> Create a work and SW_DIR directory under /pool/bbrprod in SW_DIR run: <pre> ln -s /stage/sl3-lcg-exp/babarsgm/bin bin ln -s /stage/sl3-lcg-exp/babarsgm/kanga kanga
Copy the grid release directory to /pool/bbrprod/SW_DIR
In the grid release directory run
ln -s ../kanga kanga
Create a run directory under work and copy all the files from the run directory of the test run into it.
Edit run_<run_id>.sh
to remove everything after the line:
echo copy the skimming output to the SE
and edit the tcl file to make it only run on the first 10 events.
Run the test job by running from a new login shell (i.e. one where you haven't run srtpath):
./run_<run_id>.sh
Fix and problems (preferably by adding whatever is needed into either the mk-lists.sh
or mk-tarball.pl
).
And rerun the job after cleaning up with:
rm -rf <run_id>/ skim<run_id>-01.tar
Once the job runs cleanly and saves it's output to xrootd you're ready to tar up the grid release directory, save it to the grid then install it at remote sites.
Installing the Tarball on a Grid Site
First tar up the release directory: </pre> tar -czf skim-tarball-24.2.1d.tar.gz skim-tarball-24.2.1d </pre>
Then copy that file to a grid storage element:
lcg-cr -v --vo babar -d heplnx204.pp.rl.ac.uk \ -P SPGrid/bbrgrid-soft/babar-skim-tarball-24.2.1d.tar.gz \ -l lfn:/grid/babar/SPGrid/bbrgrid-soft/babar-skim-tarball-24.2.1d.tar.gz \ file:/pool/bbrprod/skim-tarball-24.2.1d.tar.gz
Then in /stage/xrootd-data6/work/babarmc/babarsgm/SkimMiniApp/<site>
where <site>
is the grid site you want to install on edit the JDL file for the compute element you wish to use to change the version and lfn of the tarball:
cat skim-tarball-install-man-ce02.jdl Executable = "skim-tarball-install.sh"; VirtualOrganisation = "babar"; StdOutput = "install.out"; StdError = "install.out"; Environment = { "VER=skim-tarball-24.2.1f", "LFN=lfn:/grid/babar/SPGrid/bbrgrid-soft/babar-skim-tarball-24.2.1f.tar.gz", "REINSTALL=YES", "MKKANACCESS=YES" }; InputSandbox = {"../skim-tarball-install.sh","./KanAccess.cfg","./local-grid-setup","./local-grid-wrapup"}; OutputSandbox = {"install.out"}; Requirements = ( other.GlueCEUniqueID == "ce02.tier2.hep.manchester.ac.uk:2119/jobmanager-lcgpbs-babar" )
and submit the JDL file with a Software Group Manager proxy.
Chris brew 10:10, 14 Mar 2008 (GMT)