StoRM

From GridPP Wiki
Jump to: navigation, search

StoRM is a disk based storage system with a SRM interface, developed by INFN. Indeed the capitalisation of StoRM is a play on SRM. It is an increasingly used solutions for UK sites involved in LCG to provide an SRM interface to the Grid (the others being Disk Pool Manager and DCache). It offers particular advantages with respect to those systems in offering POSIX access as well as being able to operate on top of cluster filesytems such as IBM's GPFS and LUSTRE (or whamcloud verson http://www.whamcloud.com/lustre/) . This page intends to provide information for Tier-2 sites who are deploying StoRM as their SRM. The information here is intended to augment the official documentation, not replace it.

Installation

Storm can easily be installed with YAIM. Detailed documentation, including an installation guide are available in the official documentation. The pages linked below aim to offer a more basic HOWTO for new users.

  • Storm Install This is now out of date (it was written in 2009), but contains some useful information.

GridPP sites using StoRM

  • UKI-LT2-QMUL (Queen Mary, University of London).
    • QMUL is an early adopter for StoRM.
  • UKI-SOUTHGRID-SUSSEX (Sussex)

Configuration Tips

Checksums

StoRM supports checksums - and it is strongly recommended that they be enabled. Currently (StoRM 1.11.1), the same checksum algorithm must be used for all VOs - and the LHC VOs have chosen to use adler32.

  • Checksums are stored in an extended attribute of the file: user.storm.checksum.adler32
[root@se03 dteam]# getfattr -d testfile-put-1277458233-3a96016c8354.txt
# file: testfile-put-1277458233-3a96016c8354.txt
user.storm.checksum.adler32="1a400272"


Enabling checksums

StoRM's gridftp server supports calculating checksums on the fly - as the file is downloaded. To enable this, the following parameter needs to be enabled in your site-info.def:

GRIDFTP_WITH_DSI="yes"

To calculate the adler32 for a file you can use this script (adler32.py) run with python adler32.py filename


#!/usr/bin/env python
BLOCKSIZE=256*1024*1024
import sys
from zlib import adler32
for fname in sys.argv[1:]:
 asum = 1
 with open(fname) as f:
   while True:
     data = f.read(BLOCKSIZE)
     if not data:
        break
     asum = adler32(data, asum)
     if asum < 0:
       asum += 2**32
print hex(asum)[2:10].zfill(8).lower(), fname

Argus

StoRM can use Argus for authentication and user banning. Information about configuration for banning can be found here.

Implement by setting in the storm config

STORM_FE_USER_BLACKLISTING=true

gridftp isn't banned using argus - a simple patch in the ticket shows how to fix this.

Spacetokens for Atlas

All of this should end up on the StoRM website, but is here for those who may find it useful.

Note that at the time of writing (April 2012) these links to the mailing list archive no longer work. The new list server is at https://lists.infn.it/sympa/arc/storm-users , but needs a password to be setup.

  • How to set spacetoken sizes using YAIM in Storm 1.5.4

https://iris.cnaf.infn.it/pipermail/storm-users/2010-October/001025.html


Yaim's site-info.def

Files written by a atlas production user will not, by default, be readable by normal atlas users. If the SRM layer is used to access a file, this shouldn't be a problem, but sometimes ATLAS bypass this. To ensure normal ATLAS users have access to a file, STORM_TOKENNAME_DEFAULT_ACL_LIST="atlas:R" to ensure that users in the atlas group have read access to the file by default.

Previously I recommended extending this to prdatl as well - but those users are also in the atlas group, so this is not necessary.


STORM_ATLASDATADISK_VONAME=atlas
STORM_ATLASDATADISK_ACCESSPOINT=/atlas/atlasdatadisk
STORM_ATLASDATADISK_ROOT=$STORM_DEFAULT_ROOT/atlas/atlasdatadisk
STORM_ATLASDATADISK_TOKEN=ATLASDATADISK
STORM_ATLASDATADISK_ONLINE_SIZE=589000
STORM_ATLASDATADISK_DEFAULT_ACL_LIST="atlas:R"


# GROUPDISK is being incorporated into datadisk - new sites may not want to deploy this token. 
STORM_ATLASGROUPDISK_VONAME=atlas
STORM_ATLASGROUPDISK_ACCESSPOINT=/atlas/atlasgroupdisk
STORM_ATLASGROUPDISK_ROOT=$STORM_DEFAULT_ROOT/atlas/atlasgroupdisk
STORM_ATLASGROUPDISK_TOKEN=ATLASGROUPDISK
STORM_ATLASGROUPDISK_ONLINE_SIZE=285000
STORM_ATLASGROUPDISK_DEFAULT_ACL_LIST="atlas:R" 


STORM_ATLASLOCALGROUPDISK_VONAME=atlas
STORM_ATLASLOCALGROUPDISK_ACCESSPOINT=/atlas/atlaslocalgroupdisk
STORM_ATLASLOCALGROUPDISK_ROOT=$STORM_DEFAULT_ROOT/atlas/atlaslocalgroupdisk
STORM_ATLASLOCALGROUPDISK_TOKEN=ATLASLOCALGROUPDISK
STORM_ATLASLOCALGROUPDISK_ONLINE_SIZE=110000
STORM_ATLASLOCALGROUPDISK_DEFAULT_ACL_LIST="atlas:R" 
STORM_ATLASPRODDISK_VONAME=atlas
STORM_ATLASPRODDISK_ACCESSPOINT=/atlas/atlasproddisk
STORM_ATLASPRODDISK_ROOT=$STORM_DEFAULT_ROOT/atlas/atlasproddisk
STORM_ATLASPRODDISK_TOKEN=ATLASPRODDISK
STORM_ATLASPRODDISK_ONLINE_SIZE=15000

STORM_ATLASSCRATCHDISK_VONAME=atlas
STORM_ATLASSCRATCHDISK_ACCESSPOINT=/atlas/atlasscratchdisk
STORM_ATLASSCRATCHDISK_ROOT=$STORM_DEFAULT_ROOT/atlas/atlasscratchdisk
STORM_ATLASSCRATCHDISK_TOKEN=ATLASSCRATCHDISK
STORM_ATLASSCRATCHDISK_ONLINE_SIZE=60000
STORM_ATLASSCRATCHDISK_DEFAULT_ACL_LIST=atlas:R

It shouldn't be necessary to have the following space tokens: ATLASGENERATEDDISK, ATLASINSTALLDISK and ATLAS. The ATLAS token with path /atlas/notoken is necessary (and needs to be last) as a default for files that don't specify a spacetoken. ATLASINSTALLDISK and ATLASGENERATEDDISK were needed in 2011, but probably aren't any more.

STORM_ATLASGENERATEDDISK_VONAME=atlas
STORM_ATLASGENERATEDDISK_ACCESSPOINT=/atlas/generated
STORM_ATLASGENERATEDDISK_ROOT=$STORM_DEFAULT_ROOT/atlas/generated
STORM_ATLASGENERATEDDISK_TOKEN=ATLASGENERATEDDISK
STORM_ATLASGENERATEDDISK_ONLINE_SIZE=1000
STORM_ATLASINSTALLDISK_VONAME=atlas
STORM_ATLASINSTALLDISK_ACCESSPOINT=/atlas/install
STORM_ATLASINSTALLDISK_ROOT=$STORM_DEFAULT_ROOT/atlas/install
STORM_ATLASINSTALLDISK_TOKEN=ATLASINSTALLDISK
STORM_ATLASINSTALLDISK_ONLINE_SIZE=1000

STORM_ATLAS_VONAME=atlas
STORM_ATLAS_ACCESSPOINT=/atlas/atlasnotoken
STORM_ATLAS_ROOT=$STORM_DEFAULT_ROOT/atlas/atlasnotoken
#STORM_ATLAS_TOKEN=ATLAS
STORM_ATLAS_ONLINE_SIZE=1000
#STORM_ATLAS_DEFAULT_ACL_LIST=atlas:R
#Hotdisk has been decomissioned, so this is historical
# STORM_ATLASHOTDISK_VONAME=atlas
# STORM_ATLASHOTDISK_ACCESSPOINT=/atlas/atlashotdisk
# STORM_ATLASHOTDISK_ROOT=$STORM_DEFAULT_ROOT/atlas/atlashotdisk
# STORM_ATLASHOTDISK_TOKEN=ATLASHOTDISK
# STORM_ATLASHOTDISK_ONLINE_SIZE=3000
# STORM_ATLASHOTDISK_DEFAULT_ACL_LIST="atlas:R"

Restricting access

To limit writing to certain storage areas to only users with a production role, the following is used (note atlas production users are in the unix group prdatl ).

/etc/storm/backend-server/path-authz.db contains:

#################################
#  Path Authorization DataBase  #
#################################

# Evaluation algorithm
#  - possible values are:
#    1) it.grid.storm.authz.path.model.PathAuthzAlgBestMatch
#       - To determine if a request succeeds, the algorithm process 
#         the ACE entries in a computed order. Only ACEs which have 
#         a "local group" that matches the subject requester are considered.
#         The order of ACE is defined on the base of distance from StFN 
#         targetted and the Path specified within the ACE. Each ACE is 
#         processed until all of the bits of the requester's access have 
#         been checked. The result will be:
#         - NOT_APPLICABLE if there are no ACE matching with the requester.
#         - INDETERMINATE if there is at least one bit not checked.
#         - DENY if there is at least one bit DENIED for the requestor
#         - PERMIT if all the bits are PERMIT      

algorithm=it.grid.storm.authz.path.model.PathAuthzAlgBestMatch

# ==================
# SRM Operations 
# ==================
# PTP    -->   WRITE_FILE + CREATE_FILE
# RM     -->   DELETE_FILE
# MKDIR  -->   CREATE_DIRECTORY
# RMDIR  -->   DELETE
# LS     -->   LIST_DIRECTORY
# PTG    -->   READ_FILE

# ==================
# Operations on Path
# ==================
#   'W' : 	WRITE_FILE              "Write data on existing files"
#   'R' : 	READ_FILE               "Read data"
#   'F' : 	MOVE/RENAME             "Move a file"
#   'D' : 	DELETE                  "Delete a file or a directory"
#   'L' : 	LIST_DIRECTORY          "Listing a directory"
#   'M' : 	CREATE_DIRECTORY        "Create a directory"
#   'N' : 	CREATE_FILE             "Create a new file"
#

#--------+----------------------+---------------+----------
# user	 | 	      Path          |   Permission  |   ACE
# class	 |                      |   mask        |   Type
#--------+----------------------+---------------+----------
 prdatl /atlas/atlasdatadisk WRFDLMN permit
 pilatl /atlas/atlasdatadisk RL permit
 atlas  /atlas/atlasdatadisk RL permit
 @ALL@  /atlas/atlasdatadisk WRFDLMN deny
 prdatl /atlas/atlasscratchdisk WRFDLMN permit
 pilatl /atlas/atlasscratchdisk WRFDLMN permit
 atlas  /atlas/atlasscratchdisk WRFDLMN permit
 @ALL@  /atlas/atlasscratchdisk WRFDLMN deny
 prdatl /atlas/atlaslocalgroupdisk WRFDLMN permit
 pilatl /atlas/atlaslocalgroupdisk RL permit
 atlas  /atlas/atlaslocalgroupdisk RL permit
 @ALL@  /atlas/atlaslocalgroupdisk WRFDLMN deny
 prdatl /atlas/ RL permit
 pilatl /atlas/ RL permit
 atlas  /atlas/ RL permit
 @ALL@  /atlas/ WRFDLMN deny
 prdlon /vo.londongrid.ac.uk/ WRFDLMN permit
 longrid /vo.londongrid.ac.uk/ WRFDLMN permit
 @ALL@     /vo.londongrid.ac.uk/      RL permit
 @ALL@     /vo.londongrid.ac.uk/      WFDMN deny
 @ALL@     /                        WRFDLMN          permit

Operational issues

Generating a list of SURLS

It is often useful to generate a list of SURLs. For Lustre, lfs find is faster than find, and sed can then be used to turn a filename into a SURL. Here's an example for QMUL:

lfs find -type f  /mnt/lustre_0/storm_3/atlas/ | sed s%/mnt/lustre_0/storm_3/%srm://se03.esc.qmul.ac.uk/%

In the case of a disk server that is down:

 lfs df 

will tell you which OSTs are down.

lfs find -obd lustre_0-OST002f_UUID  /mnt/lustre_0/storm_3/atlas/ | sed s%/mnt/lustre_0/storm_3/%srm://se03.esc.qmul.ac.uk/%
 

will find files on a particular OST: lustre_0-OST002f_UUID in this case.

Syncat Dumps

Christopher Walker has a very alpha quality syncat dump script which is available on request.



Hardware

The official documentation has some information on hardware requirements, but it may be useful to know what sites currently have deployed - so here are some examples. Note that this is not a statement of what is actually required.

StoRM's hardware requirements (as opposed to the underlying GPFS/Lustre filesystem) are modest - the main point is that the GridFTP servers need lots of bandwidth.

QMUL

QMUL runs the frontend, backend and database on one machine.

Hardware config as of April 2017 :

Dell R510 * 72
CPU: Dual X5650 
Memory: 24Gig RAM
DISK: 12 * 2(3)TB in Raid 6
Network: 10Gig connectivity.
Dell R730XD * 20
CPU: Dual E5 2609 V3 
Memory: 64Gig RAM
DISK: 16 * 6TB in Raid 6
Network: 10Gig connectivity.
HPE APOLLO 4200 * 2
CPU: Dual E5 2609 V3 
Memory: 128Gig RAM
DISK: 14 * 8TB in 2 * Raid 6
Network: 10Gig connectivity.


Note, we updated the driver for this from the SL6.7 default to the latest on the Intel website after suffering some hangs.

QMUL also has a second GridFTP server which runs on similar hardware - initially this was deployed to make use of a second college link. It is currently being used to provide a bit of extra performance and test jumbo frames.

This page is a Key Document, and is the responsibility of Dan Traynor. It was last reviewed on 2017-07-05 when it was considered to be 70% complete. It was last judged to be accurate on 2014-10-02.