DPM Fuse

From GridPP Wiki
Jump to: navigation, search

Intro

UPDATE May 2013: Doesn't time fly. Now there is a new fuse module so previous instructions are moved below. UPDATE- 2 March 2014: Now you can use davix to mount http spaces.

Installing

See https://svnweb.cern.ch/trac/lcgutil/wiki/gfalfs

yum install gfal2-all gfalFS
usermod -a -G fuse wbhimji

From epel-testing at present

yum install gfal2-plugin-xrootd 
gfalFS ~/mount_tmp root://srm.glite.ecdf.ed.ac.uk/dpm/ecdf.ed.ac.uk/home/atlas

Sometimes there is an error the first time a directory is listed after mounting. But it seems never to reappear

[wbhimji@gridpp09 ~]$ ls ~/mount_tmp/
ls: reading directory /home/wbhimji/mount_tmp/: Inappropriate ioctl for device
[wbhimji@gridpp09 ~]$ ls ~/mount_tmp/
AOD.067184.big.pool.root

Using with Davix

yum install gfal2-all gfalFS gfal2-plugin-http 
usermod -a -G fuse wbhimji 
/usr/bin/gfalFS /tmp/mnt2 davs://srm.glite.ecdf.ed.ac.uk//dpm//
[wbhimji@gridpp09 ~]$ find /tmp/mnt2/atlasscratchdisk/ | grep -v rucio | grep -v SAM | wc -l
6437

 time rmdir /tmp/mnt2/atlasscratchdisk/!(rucio|SAM)

BE VERY CAREFUL AS rmdir WILL REMOVE NONEMPTY DIRECTORIES.

Testing

reading file in ROOT

Reading 100% of the file with TTreeCache on (ReadTotal = 533.54 MBytes; ReadCalls = 17; EVENTS=7929)

FILENAME='/home/wbhimji/mount_tmp/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root' WALLTIME=39.4747 CPUTIME=33.42

Didn't clear the cache then ran again: FILENAME='/home/wbhimji/mount_tmp/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root' WALLTIME=38.7424 CPUTIME=33.53

Did clear the cache then running with xrootd directly FILENAME='root://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root' WALLTIME=31.5409 CPUTIME=31.72

and again FILENAME='root://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root' WALLTIME=33.2425 CPUTIME=33.66

Reading 100% of the file with TTreeCache off (ReadCalls = 115445)

FILENAME='root://srm.glite.ecdf.ed.ac.uk//dpm/ecdf.ed.ac.uk/home/atlas/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root' WALLTIME=34.027 CPUTIME=24.66 WALLTIME=35.5738 CPUTIME=25.29

FILENAME='/home/wbhimji/mount_tmp/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root' WALLTIME=16.1109 CPUTIME=14.98

FILENAME='/home/wbhimji/mount_tmp/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root' WALLTIME=15.1924 CPUTIME=14.05

[wbhimji@gridpp09 ROOTIOTests]$ ./releaseFileCache ~/mount_tmp/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root [wbhimji@gridpp09 ROOTIOTests]$ ./readDirect /home/wbhimji/mount_tmp/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.4/group.test.hc.NTUP_SMWZ-Root530.root physics 10 0 WALLTIME=16.9439 CPUTIME=15.85

Athena

Ran

Reco_trf.py inputAODFile=/home/wbhimji/mount_tmp/atlasdatadisk/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.AOD.root outputNTUP_TOPFile=topAlldata3.root maxEvents=10

it ran fine - performance comparison with all events to follow.

Previous fuse module experience

Jean-Phillipe once wrote a gfal Fuse module. Rod sent it to me - and I tested it with DPM below.

Installing

Basic instructions to get it to work

yum install fuse-libs fuse-devel fuse 
gcc -Wall `pkg-config fuse --cflags --libs` -I/opt/lcg/include -L/opt/lcg/lib64 -lgfal gfalfs.c
fusermount -u /tmd/fuse
./a.out -d -o kernel_cache /tmp/fuse >& FuseLog.log & 

The -d gives a lot of useful debug output for testing

Make sure gfalfs.c contains the line:

static const char *gfal_data_access_protocol = "rfio:////dpm/"

with all 4 slashes - or you will get authentication issues.

Testing

cp

A simple test of copying a file.

vm008:~/wahidtest# time rfcp /dpm/gla.scotgrid.ac.uk/home/atlas/atlasmcdisk/mc09_7TeV/AOD/e521_s765_s767_r1250_r1260/mc09_7TeV.106059.PythiaB_ccmu15X.merge.AOD.e521_s765_s767_r1250_r1260_tid129608_00/AOD.129608._000145.pool.root.1 /tmp/AOD.root
2913440544 bytes in 38 seconds through eth1 (in) and local (out) (74872 KB/sec)
real    0m40.611s
user    0m0.364s
sys     0m17.401s


vm008:~/wahidtest# time cp /tmp/fuse/gla.scotgrid.ac.uk/home/atlas/atlasmcdisk/mc09_7TeV/AOD/e521_s765_s767_r1250_r1260/mc09_7TeV.106059.PythiaB_ccmu15X.merge.AOD.e521_s765_s767_r1250_r1260_tid129608_00/AOD.129608._000145.pool.root.1 /tmp/AOD.root
real    1m13.030s
user    0m0.092s
sys     0m4.247s


Atlas AOD reading in ROOT

This test (documented at Local_File_Access) runs through the entries of a root tree - an atlas AOD file was chosen. TTreeCache was switched off for these tests. Turning it on would improve performance. Both jobs: ReadTotal = 2710.43 MBytes ReadCalls = 3195026

Fuse:

Percent of CPU this job got: 64%
Elapsed (wall clock) time (h:mm:ss or m:ss): 11:56.25
Disk IO   =  24.501 MBytes/s

Rfio

Percent of CPU this job got: 71%
Elapsed (wall clock) time (h:mm:ss or m:ss): 11:22.74
Disk IO   =  35.136 MBytes/s

athena

a relevant bug

Running D3PDMaker (or any ReExCommon analysis (or probably all athena jobs now) segfaults. This seems to be because the same file is opened twice (first for the InputFilePeeker then for the real job).

(gdb) bt
#0  0xf6165b4b in rfio_close_v2 (s=6) at close.c:146
#1  0xf61656f6 in rfio_close (s=6) at close.c:41

Jean-Philippe says "there is a problem with gfal when doing two opens. The problem seems to be in the security layer but is not understood yet." He could pursue this further if it needs to be solved


For testing we can get around this by opening the file over rfio for the file peeker then fuse the second time i.e jobOptions:

athenaCommonFlags.FilesInput= ['rfio:/dpm/gla.scotgrid.ac.uk/home....']
and after include ("./RecExCommon_topOptions.py") :
svcMgr.EventSelector.InputCollections = ["/tmp/fuse/gla.scotgrid.ac.uk/home/atlas/.../AOD.129608._000150.pool.root.1"]   

some test results

On reordered files with or without TTreeCache (TC).

Release 15.9.0 has fix for ROOT to allow Rfio vector read and all tests done with dpm 1.7.4-7 client libs (which has fix for TC/vreads)

athena perf mon reported time / evt (ms) (CPU eff (from time)):

Test Rfio Fuse
15.6.10 No TC 761 (59%) 457 (65%)
15.9.0 TC 602 (64%) 447ms (70%)

Hammercloud