RAL Tier1 DCache Operational Procedures

This page is for documenting procedures followed at the Tier 1, and as such is primarily of interest to Tier 1 sysadmins

What's where?

Where to find the dCache systems, the name of the dcache related services running on these systems, other useful information for Tier 1 sysadmins who are not interacting with dCache on a daily basis, but may have to in an emergency.

Disk servers

Physical Location

Disk Servers are in both A1 Upper and A5 Lower.

Services

The dcache pools are controlled by the /etc/init.d service dcache-pool

Network Connections

TBD

Data Transfer systems

Physical Location

All gftp system are in A5 Lower, third rack of Compusys 2004 cpu nodes, lower half.

Services

The dcache gridftp and gsidcap door are controlled by the /etc/init.d service dcache-core

Network Connections

TBD

SRM systems

We have two SRM systems, one intended for processing requests for disk files (dcache.gridpp.rl.ac.uk), another for processing requests for tape files (dcache-tape.gridpp.rl.ac.uk). This distinction is somewhat artificial as both srm can access all files stored in dCache, which can be useful in determining if a fault is with a particular SRM, or the entire system.

Physical Location

Both SRM systems are in A5 Lower, third rack of Compusys 2004 cpu nodes, lower half.

Services

The dcache gridftp and gsidcap door are controlled by the /etc/init.d service dcache-core

Network Connections

TBD

Central systems

There are two systems which are critical to the running of dCache. dcache-head.gridpp.rl.ac.uk is the central administration system, it runss the ssh admin interface, the monitoring web pages, the module which select which pool to use for a data transfer and several others. pnfs.gridpp.rl.ac.uk runs a postgresql database, this contains the mapping from the /pnfs paths to the internal ids, this is used by the pnfs service. The system also runs a PnfsManager dCache cell to interface dCache and pnfs together.

Physical Location

dcache-head.gridpp.rl.ac.uk is in A5 Lower, third rack of Compusys 2004 cpu nodes, lower half. pnfs.gridpp.rl.ac.uk is in A1 Upper, second rack of ClusterVision 2003 cpu nodes.

Services

The services on dcache-head are controlled by the /etc/init.d service dcache-core The service on pnfs are controlled by the /etc/init.d services postgresql pnfs & dcache-core

Network Connections

TBD

Miscellaneous systems

pg350 lcg0438

Adding new things

Adding a pool

Adding a pool to an existing dCache server

These instructions assume that the server is already setup to run dCache and that you are adding space to an existing vo.

• In the new allocated space, create a pool directory and inside the pool directory create a control and data directory:

   pool
   \- control
   \- data

• Create a setup file into the pool directory, the simplest way is to copy one from an existing pool - experience has shown that the pools on a server tend to be of similar size; if not, edit the

  set max diskspace XXXXX

line in the setup file to an appropriate figure.

• Add the pool to the /opt/d-cache/config/<shorthostname>.poollist file with an entry like:

  vo_1  /path/to/pool  sticky=allowed recover-space recover-control recover-anyway lfs=precious tag.hostname=<host>

Again adapting an existing entry is the easiest way to proceed. Current convention is to use the name of the vo followed by an underscore followed by the data partition number - see the output of the mount command. Check there are no blank lines at the end of the file after you've finished editing, otherwise the next step will throw out odd errors.

• Restart the dcache-pool service, The pools log into /var/log/<shorthostname>Domain.log, note that it can take some time for the existing pools to start up if they are filled with files.

• Login to the dcache ssh admin service on dcache-head.gridpp.rl.ac.uk and cd PoolManager to change in to the PoolManager setup, and type the following for each vo that pool is being made usable for (normally only one)

 psu addto pgroup <vo>-pgroup <poolname>
 save

The space should now be available to the vo. However they will not be aware of the fact until the information system is updated

Adding a new server

Copying files from existing servers is the easiest way to deploy a new server. This doesn't set up pools properly - see Adding Pools for that

Network Setup

(Possibly optional if host will not be accepting traffic over UKLight - mainly non-LHC experiments)

• Get UKLight visible IP address
• Change ip address (/etc/hosts, /etc/sysconfig/network-scripts/ifcfg-eth0)
• Request DNS change
• Request SURE update
• Configure UKLight routes (CERN and Lancaster) (tier1-routes-config rpm)

System Setup

• Chown array mount points to appropriate ownership - for automated stats gathering
• Add logrotate for pool logfile (/var/log/<shorthostname>Domain.log)
• Change ganglia cluster (/etc/gmond.conf)

Software Setup

Comment out any lines in /etc/exports and run exportfs -av

Add these lines to /etc/yum.conf

[dcache]
name=Dcache packages for RedHat $releasever
baseurl=http://touch.gridpp.rl.ac.uk/yum/dcache/$releasever/

[cacerts]
name=LCG CA Certificates
baseurl=http://touch.gridpp.rl.ac.uk/yum/lcg/ca/

[lcg]
name=LCG packages
baseurl=http://wwwinstall.gridpp.rl.ac.uk/yum/lcg/2_3_0/rh73/

• Do yum install dcache-server j2dsk lcg-CA
• Put certificate/key in /etc/grid-security/
• Configure /opt/d-cache/config/dCacheSetup and pool.batch
• Copy /opt/d-cache/etc/node_config from an existing server
• touch /opt/d-cache/etc/pool_path
• Run /opt/d-cache/install/install.sh
• Symlink /opt/d-cache/bin/dcache-pool to /etc/init.d/dcache-pool
• chkconfig --add dcache-pool
• chkconfig --on dcache-pool
• service dcache-pool start

Adding a vo

Setting up user mappings

The disk servers are hooked into the Tier 1 nis setup, so should get updates for any pool accounts and groups automatically and don't need a grid-map file or dcache.kpwd file The gftp servers, dcache, dcache-tape and pnfs are not and will need the accounts added. Ensure that the lcg-yaim and tier1-yaim-config rpms are installed and up to date, then on each host run :

 /opt/lcg/yaim/scripts/run_function /opt/lcg/yaim/etc/site-info.def config_users config_mkgridmap

Note that dcache.gridpp.rl.ac.uk and dcache-tape.gridpp.rl.ac.uk, because they run information providers, have special cases for the VOS variables in the site-info.def file and may not have been updated at the same time, see the Information system integration heading further down for how to build the rpm in this case. This should create the necessary group and user accounts and add the correct lines to /opt/edg/etc/edg-mkgridmap.conf. Then run the /opt/d-cache/bin/grid-mapfile2dcache-kpwd script to update the /opt/d-cache/etc/dcache.kpwd file.

Pnfs setup

On pnfs.gridpp.rl.ac.uk, Run

 /opt/pnfs/tools/mdb show 

and compare the number of databases displayed with the number of shmservers in the /usr/etc/pnfsSetup file. If necessary increase the number of shmservers in the pnfsSetup file and restart the pnfs service.

If the directory you wish to create exists already then run /opt/pnfs/tools/pathfinder <directory excluding trailing slash>, if the first line of output begins with 0001000.... <directory> then directory was probably automatically created by the /opt/d-cache/bin/gridmapfile2dcachekpwd script and, if empty, should be removed.

Follow the instructions in the DCache documentation for creating a new pnfs database, however note that convention is when adding the directory tags to use the name of the vo in place of both myStore and STRING (unless adding a VO for tape access in which case STRING is replaced by tape).

Then chown the vo's directory to the <vo>001 account and the <vo> group account, and then chmod the vo directory to be group writeable.

Pnfs Database Replication

TBD

PoolManager configuration

On dcache-head.gridpp.rl.ac.uk, Log into the admin interface and cd into the PoolManager module and type

 psu create pgroup <vo>-pgroup
 psu create unit -store <vo>:<vo>@osm
 psu create ugroup <vo>
 psu addto ugroup <vo> <vo>:<vo>@osm
 psu create link <vo>-link minos
 psu set link <vo>-link -readpref=10 -writepref=10 -cachepref=10
 psu add link <vo>-link <vo>-pgroup
 save

Information System integration

Currently its necessary to copy the YAIM config_gip function to /opt/lcg/yaim/functions/local/config_gip and comment out the last section about configuring gin as dCache does not yet integrate with RGMA.

Add new vos by getting the latest src rpm from touch.gridpp.rl.ac.uk /kickstart/yum/local/sl3/, and updating the site-info.def in that and releasing a new rpm.

• Edit the file /var/lib/edginfo/available-space.dat to include the total space for the new vo
• Edit the file /var/lib/edginfo/used-space.dat to include the used space for the new vo
• Edit the /opt/lcg/libexec/lcg-info-dynamic-dcache script to add the new vo
• If adding new disk space, run the following command on dcache.gridpp.rl.ac.uk,

  ./run_function ../etc/site-info.def config_gip

• If adding new tape storage, on dcache-tape.gridpp.rl.ac.uk
  • Ensure that the paths generated in the config_gip local yaim function are updated to be "tape" instead of "data"
  • Then run :

  ./run_function ../etc/site-info.def config_gip

HSM Interface

The current HSM interface pools are

• All tape enabled vos (except lhcb)
  • csfnfs60_1
  • csfnfs61_1
  • csfnfs62_1
  • csfnfs63_1
• lhcb tape pools
  • lhcb_57_1
  • lhcb_57_2
  • lhcb_57_3
  • lhcb_57_4

Creating an HSM pool

Initial setup is broadly similar to Adding a Pool above with some changes detailed here.

Sme entries need to be added to the setup file

   hsm set osm -pnfs=/pnfs/gridpp.rl.ac.uk/
   hsm set osm -command=/root/pnfs2ads2.sh

The pnfs2ads2.sh script should be copied from another hsm pool server or the Storage Group CVS repository for dcache-stager. Queue definitions must also be added to the setup file. While possible to have a pool solely dedicated to one VOs tape access, all HSM pools deployed to date allow all tape enabled VOs

   queue define class osm lhcb:tape -pending=0 -total=0 -expire=0
   queue define class osm minos:tape -pending=0 -total=0 -expire=0
   queue define class osm dteam:tape -pending=20 -total=0 -expire=0
   queue define class osm atlas:tape -pending=0 -total=0 -expire=0
   queue define class osm cms:tape -pending=0 -total=0 -expire=0

The /opt/d-cache/config/<shorthostname>.poollist entry for an hsm pool is slightly shorter than that for a non-hsm pool, the lfs=precious is ommitted

   vo_1  /path/to/pool sticky=allowed recover-space recover-control recover-anyway tag.hostname=<host>

The dcache-pool service should now be (re)started and the pool should appear. The names of the pool groups to add the new hsm pool to are by convention <vo>-tape-pgroup.

Disabling HSM Interface

For each pool, in its admin interface, run :

 queue suspend class *

Reenabling HSM Interface

For each pool, in its admin interface, run :

 queue resume class *

Answering Questions

This section deals with techniques to answer questions that are asked about the dCache service. Some of these involve interacting with the dcache admin interface, currently this is only accessible from dcache-head.gridpp.rl.ac.uk, ask Derek for access if you can't ssh to that system. See the dCache introduction to admin interface for general information about using the admin interface.

File X apparently exists but cannot be read from

Cause 1: This can be caused by the file being in the /pnfs filesystem but not on an working pool

In the admin interface , change into the PnfsManager cell and do:

 cacheinfoof <filename>

This should return a list of pools that dCache believes the files is on:

 (PnfsManager) admin > cacheinfoof /pnfs/gridpp.rl.ac.uk/data/dteam/dr35/20060512-1258
 csfnfs62_4

If dCache does not believe the file is on any pool a blank line will be printed :

 (PnfsManager) admin > cacheinfoof /pnfs/gridpp.rl.ac.uk/data/dteam/dr35/nonexistent
       
 (PnfsManager) admin >

Notes:

• A file existing in /pnfs, but not on a pool, and a file not in /pnfs will give the same blank line output
• dCache caches information about the files stored on a pool even after the pool itself has been shutdown

Cause 2: The pool(s) that the file is on are busy

If cacheinfoof as in the section above, shows that the file is on an available pool or pools, then it is possible that a pool has queued the transfer. To proceed further we need to know the pnfsid of the file, so in the PnfsManager cell do:

  pnfsidof <filename>

This will return a hexadecimal string. Then for each pool mentioned in the output of cacheinfoof, go into its cell and run

  mover ls

Look for the pnfsid in the output.

What files are stored in pool X

If a dCache pools suffers a catastrophic failure, e.g. loses 2 disk from a RAID 5 array, then the list of files stored on that pool needs to be recovered. Depending on the type of failure it may not be possible to access the system. dCache's list of files stored on a pool can be accessed through the companion database on the pnfs host See Checking pool migration for a query that can be adapted