A quick guide to CVMFS
Contents
Deploying software with CVMFS
For more information about CVMFS at RAL, click here.
Overview of the process
- Prepare your working area
- Prepare your software
- Upload your software to the RAL CVMFS repository and deploy it
- Wait a bit...
- Run your CVMFS-powered jobs
A trivial example
Preparing your working area
Log in to your machine of choice and create a new working area.
$ ssh -Y whyntie@heppc402 whyntie@heppc402's password: # enter your password here $ mkdir cvmfstests $ cd cvmfstests $ mkdir helloworld $ cd helloworld $ pwd /users/whyntie/cvmfstests/helloworld
Preparing your software
Create a new directory that will form the basis of your CVMFS tarball.
$ mkdir hello-world_001-00-00 $ cd hello-world_001-00-00 $ pwd /users/whyntie/cvmfstests/helloworld/hello-world_001-00-00
Create three files in this directory:
-
hello-world.sh
- the "software"; -
run.sh
- the script that runs the software; -
README.md
- for your notes, ideally in the MarkDown format.
$ cat hello-world.sh #!/bin/bash # #============================================================================= # The GridPP CVMFS Hello World! Script #============================================================================= # # Usage: . hello-world.sh [whoever you want to greet] # echo 'Hello' $1'!' $ $ $ cat run.sh #!/bin/bash # #============================================================================= # The GridPP CVMFS Hello World! run script #============================================================================= # # Usage: . run.sh [full path of the hello-world.sh script] # # Note that for grid jobs (or clusters with CVMFS enabled) this will be # the CVMFS directory. # $1/hello-world.sh World $ $ $ cat README.md My CVMFS Test Notes =================== You're keeping detailed notes, right? Good good.
Don't forget to change the permissions on the scripts so that they can be run.
$ chmod a+x run.sh $ chmod a+x hello-world.sh
Now compress these files into a tarball.
$ cd ../ $ tar -czf hello-world_001-00-00.tar hello-world_001-00-00 $ ls hello-world_001-00-00 hello-world_001-00-00.tgz
Uploading and deploying the software
Create a proxy with your Virtual Organisation (VO):
$ voms-proxy-init --voms cernatschool.org
Then copy the tar ball to your CVMFS repository with the gsiscp
command:
$ gsiscp -P 1975 hello-world_001-00-00.tgz cvmfs-upload01.gridpp.rl.ac.uk:./cvmfs_repo/.
You should then be able to log on and decompress the tar ball into the repository. Your proxy will take care of the username and password.
$ gsissh -p 1975 cvmfs-upload01.gridpp.rl.ac.uk $ tar -xvf hello-world_001-00-00.tgz
Note: it may take up to three hours for your software to appear, depending on the timing of the cron jobs.
Run a job using your CVMFS software
Jobs using your software can be submitted and run as normal - but now you don't need to worry about installing software anywhere. The executables you need will be available in the CVMFS repository. So a JDL file for the Hello World! software (deployed on the cernatschool.org
repository would look like this:
$ cat helloworld.jdl # The GridPP CVMFS Hello World! JDL file Executable = "/bin/sh"; # Replace "cernatschool.gridpp.ac.uk" with your VO's address, of course! Arguments = "/cvmfs/cernatschool.gridpp.ac.uk/hello-world_001-00-00/run.sh /cvmfs/cernatschool.gridpp.ac.uk/hello-world_001-00-00"; StdOutput = "stdout.txt"; StdError = "stderr.txt"; OutputSandbox = {"stdout.txt", "stderr.txt"}; #
As you can see, the shell executable is being run with two arguments:
- The script to run -
run.sh
- which is in the tarball. - The first argument supplied to
run.sh
. In this example, the first argument ofrun.sh
base directory (including the CVMFS prefix). We have made this an argument in therun.sh
to make local testing easier - which, when the software gets more complicated, is a worthwhile thing to do...
So, if you're using glite
for your WMS, you can submit the job as usual:
$ voms-proxy-init --voms cernatschool.org Enter GRID pass phrase for this identity: # you know what to do... ...[proxy confirmation messages] $ myproxy-init -d -n Your identity: /C=UK/O=eScience/OU=QueenMaryLondon/L=Physics/CN=tom whyntie Enter GRID pass phrase for this identity: # and again... ...[proxy confirmation messages] $ glite-wms-job-submit -a -o jobIDfile helloworld.jdl ...[job submission messages]
When you retrive the output from the jobs, stdout.txt
should contain a very, very exciting message, indicating that your job has succeeded and the CVMFS software has been successfully deployed.
Useful links
Internal
External
- The CERN CVMFS page;
- CernVM-FS - Building an Infrastructure for Non-LHC Computing - ISGC 2014 talk from I. Collier (slides).