A quick guide to CVMFS

Deploying software with CVMFS

For more information about CVMFS at RAL, click here.

Overview of the process

1. Prepare your working area
2. Prepare your software
3. Upload your software to the RAL CVMFS repository and deploy it
4. Wait a bit...
5. Run your CVMFS-powered jobs

A trivial example

Preparing your working area

Log in to your machine of choice and create a new working area.

$ ssh -Y whyntie@heppc402
whyntie@heppc402's password: # enter your password here

$ mkdir cvmfstests
$ cd cvmfstests
$ mkdir helloworld
$ cd helloworld
$ pwd
/users/whyntie/cvmfstests/helloworld

Preparing your software

Create a new directory that will form the basis of your CVMFS tarball.

$ mkdir hello-world_001-00-00
$ cd hello-world_001-00-00
$ pwd
/users/whyntie/cvmfstests/helloworld/hello-world_001-00-00

Create three files in this directory:

• hello-world.sh - the "software";
• run.sh - the script that runs the software;
• README.md - for your notes, ideally in the MarkDown format.

$ cat hello-world.sh 
#!/bin/bash
#
#=============================================================================
#                     The GridPP CVMFS Hello World! Script
#=============================================================================
#
# Usage: . hello-world.sh [whoever you want to greet]
#
echo 'Hello' $1'!'
$
$
$ cat run.sh 
#!/bin/bash
#
#=============================================================================
#                    The GridPP CVMFS Hello World! run script
#=============================================================================
#
# Usage: . run.sh [full path of the hello-world.sh script]
#
# Note that for grid jobs (or clusters with CVMFS enabled) this will be
# the CVMFS directory.
#
$1/hello-world.sh World
$
$
$ cat README.md
My CVMFS Test Notes
===================
You're keeping detailed notes, right? Good good.

Don't forget to change the permissions on the scripts so that they can be run.

$ chmod a+x run.sh
$ chmod a+x hello-world.sh

Now compress these files into a tarball.

$ cd ../
$ tar -czf hello-world_001-00-00.tar hello-world_001-00-00
$ ls
hello-world_001-00-00 hello-world_001-00-00.tgz

Uploading and deploying the software

Create a proxy with your Virtual Organisation (VO):

$ voms-proxy-init --voms cernatschool.org

Then copy the tar ball to your CVMFS repository with the gsiscp command:

$ gsiscp -P 1975 hello-world_001-00-00.tgz cvmfs-upload01.gridpp.rl.ac.uk:./cvmfs_repo/.

You should then be able to log on and decompress the tar ball into the repository. Your proxy will take care of the username and password.

$ gsissh -p 1975 cvmfs-upload01.gridpp.rl.ac.uk
$ tar -xvf hello-world_001-00-00.tgz

Note: it may take up to three hours for your software to appear, depending on the timing of the cron jobs.

Run a job using your CVMFS software

Jobs using your software can be submitted and run as normal - but now you don't need to worry about installing software anywhere. The executables you need will be available in the CVMFS repository. So a JDL file for the Hello World! software (deployed on the cernatschool.org repository would look like this:

$ cat helloworld.jdl
# The GridPP CVMFS Hello World! JDL file
Executable = "/bin/sh";
# Replace "cernatschool.gridpp.ac.uk" with your VO's address, of course!
Arguments = "/cvmfs/cernatschool.gridpp.ac.uk/hello-world_001-00-00/run.sh /cvmfs/cernatschool.gridpp.ac.uk/hello-world_001-00-00";
StdOutput = "stdout.txt";
StdError = "stderr.txt";
OutputSandbox = {"stdout.txt", "stderr.txt"};
#

As you can see, the shell executable is being run with two arguments:

1. The script to run - run.sh - which is in the tarball.
2. The first argument supplied to run.sh. In this example, the first argument of run.sh base directory (including the CVMFS prefix). We have made this an argument in the run.sh to make local testing easier - which, when the software gets more complicated, is a worthwhile thing to do...

So, if you're using glite for your WMS, you can submit the job as usual:

$ voms-proxy-init --voms cernatschool.org
Enter GRID pass phrase for this identity: # you know what to do...
...[proxy confirmation messages]
$ myproxy-init -d -n
Your identity: /C=UK/O=eScience/OU=QueenMaryLondon/L=Physics/CN=tom whyntie
Enter GRID pass phrase for this identity: # and again...
...[proxy confirmation messages]
$ glite-wms-job-submit -a -o jobIDfile helloworld.jdl
...[job submission messages]

When you retrive the output from the jobs, stdout.txt should contain a very, very exciting message, indicating that your job has succeeded and the CVMFS software has been successfully deployed.

Useful links

Internal

• CVMFS Use Cases for GridPP.

External

• The CERN CVMFS page;
• CernVM-FS - Building an Infrastructure for Non-LHC Computing - ISGC 2014 talk from I. Collier (slides).