Grid Storage
From GridPPwiki
Welcome to the GridPP Storage Resource Management wiki
This wiki is maintained by the storage community, specifically to help support UK Tier-2 sites as part of the GridPP (http://www.gridpp.ac.uk/) project. It may also contain things of interest to end users of the LHC Computing Grid (http://lcg.web.cern.ch/LCG/), and anyone else interested in storage resource management issues in grids. The deployment of an SRM is essential for all Tier-2 sites involved in LCG (http://lcg.web.cern.ch/LCG/).
If you've arrived here by happy accident, it's all about managing the huge amounts of data which will be produced by the Large Hadron Collider (http://public.web.cern.ch/Public/Content/Chapters/AboutCERN/CERNFuture/WhatLHC/WhatLHC-en.html) at CERN (http://public.web.cern.ch/Public/Welcome.html).
Start here if you are in GridPP, and you have an SRM, or want to have one, or think you might possibly want to have one.
| Table of contents |
Grid(PP) Storage Mailing Lists
Then we highly recommend that you join the GridPP storage community GridPP Storage mailing list (http://www.jiscmail.ac.uk/lists/gridpp-storage.html). You can freely browse the archives, but must be a member to post. The page contains details how to join. Most of the support is via the mailing list and we have a good friendly community with a lot of experience.
Other useful mailing lists include GridPP service challenge (http://www.jiscmail.ac.uk/lists/gridpp-sc.html) for service challenges and more generally data transfers from T1->T2, and CASTOR support (http://www.jiscmail.ac.uk/lists/CASTOR-SUPPORT.html) for CASTOR-at-RAL discussions and minutiae. For those just interested in CASTOR announcement (downtime etc), we have the CASTOR announce (http://listserv.cclrc.ac.uk/lists/CASTORPP-L.html) list.
Weekly meetings
You are also welcome to join the (currently) weekly conferences, every Wednesday 1000-1030 (BST). Details are announced to the mailing list. We now use EVO (http://evo.caltech.edu/evoGate/) for the meetings, and the agenda for each meeting is available on the UKI-ROC (http://indico.cern.ch/categoryDisplay.py?categId=338) page.
Storage blog
In addition to the mailing list, storage related news items are regularly posted on the GridPP Storage blog (http://gridpp-storage.blogspot.com). You can subscribe to the RSS feed here (http://gridpp-storage.blogspot.com/rss.xml).
Bug tracker
You can also view the current bug list in the Support project page (http://savannah.cern.ch/projects/srmsupportuk). You have to have an account at CERN Savannah to be able to post and edit bugs. If you are deploying SRMs, we recommend that you do that.
The expected level and timescale of support that GridPP will provide to the UK Tier-2 sites was described in a 2005 document Tier-2 Support Document (http://www.gridpp.ac.uk/deployment/admin/tier2-support-plan.pdf).
Storage Accounting
- Old GridPP Storage Accounting.
- New EGEE storage accounting system (http://stoac.grid-support.ac.uk/) run by NGS.
- Storage accounting consistency checklist. Tracks the consistency of the Storage Accounting system with the actual level of resources at GridPP sites. Issues and how they were resolved are recorded.
Storage Monitoring
GridPP monitors the levels of storage resources used at the Tier-1 and Tier-2 sites. Current usage and historical data can be found on the status page (http://www.gridpp.ac.uk/storage/status/gridppDiscStatus.html).
Individual SEs are monitored on the GOC SE monitoring (http://goc02.grid-support.ac.uk/cgi-bin/srm.py) page. And now also on the Service Availability Monitoring (SAM) tests [1] (https://lcg-sam.cern.ch:8443/sam/sam.py).
See also the Storage Monitoring and Accounting overview page.
Storage Resource Management (SRM)
SRM (Storage Resource Manager) is a protocol for Grid access to mass storage systems (tape or disk or disk arrays). The protocol itself is a collaboration (http://sdm.lbl.gov/srm-wg/) between Lawrence Berkeley (LBNL), Fermilab (FNAL), Jefferson (JLAB), CERN, and RAL. It is also a GGF working group, GSM-WG (https://forge.gridforum.org/projects/gsm-wg/).
Sometimes, a Storage Element that supports an SRM protocol is called "an SRM".
Before starting out trying to deploy an SRM at your Tier-2 site, it is useful to know some more about SRM in general. One aspect of the protocol that will be unfamiliar to new users is the possibility for different SRM file types to exist. Also potentially confusing are the many different filenames.
Supported SRMs
Storage Hardware
Information Systems
All SRM's must publish to the Information System using the Glue SE Schema.
Operations
Operational proceedures for grid storage:
- SRM File Loss describes what to do when you lose files from your SRM system
- SE Shutdown describes how to announce the shutdown of an SRM
- SE Full describes how you should watch your SE filling up and what you should do if it happens.
- SE Lost Disk-Server describes what a site needs to do if it loses all data from a disk server.
New Storage Evaluations
- HadoopBeStMan BeStMan and Hadoop
- HadoopStorm Storm and Hadoop
Related Pages
- OSG storage and data management support and operations (http://twiki.grid.iu.edu/twiki/bin/view/Storage/)
