- 1 Grid Storage
- 2 Information about data storage and management in GridPP
- 3 GridPP expertise
GridPP runs a large scale data e-infrastructure that delivers services not just to the LHC experiments in WLCG but also to other user communities that have been approved by the PMB. Moreover, the expertise that GridPP has gained with data processing at the petabyte scale can benefit other areas of research as well.
Data in GridPP
The WLCG collaboration moves and stores hundreds of petabytes of data; the most important data is replicated globally, to ensure high availability and durability (i.e. to minimise the risk of losing it). GridPP provides storage in WLCG via the Tier 1 at RAL and via its Tier 2 sites throughout the UK. The grid storage is based on the so-called "storage elements" (or SEs), which provide a grid interface to quite diverse storage systems. SEs provide control interfaces like SRM, information interfaces based on LDAP (more or less), and interfaces for data transfer and data access such as xroot or GridFTP.
GridPP Storage and Data Management Group
Or "GridPP storage" for short.
The GridPP project runs a Storage and Data Management group, a group of experts and system administrators who support the data infrastructure: the SEs themselves and the data transfer tools and protocols, including improving performance and resilience.
Membership of the group is open to all WLCG Tier 2s and others working in grid storage including representatives from the VOs: membership can be requested at jiscmail. Indeed, other than the UK, about 10 different countries are represented on the list. Archives are public.
The group runs a weekly audio/video conference meeting every Wednesday 10.00 (UK) for its members and occasional guests. Notes from these meetings (and how to connect) can be found here; like the mailing list archives, they are also public. There is also usually a summary at the operations page.
Information about data storage and management in GridPP
Information for users (or prospective users)
Users of GridPP resources tend to be grouped together in collaborations (known as Virtual Organisations, or VOs), with a common research purpose. VOs can range in size from in principle a single person to several thousand. The Main Page has more information about "joining the grid" as a new VO or becoming a member of an existing VO.
As regards the data volume, sites can often at their discretion allocate resources as long as, say, only a few tens of terabytes are required. Experience has shown, however, that even small VOs can suddenly grow and will need stricter data management, so it is best to plan for growth by requesting - and getting - and using! - a space allocation (identified by a so-called space token.)
For "small" VOs (by data volume) it is considered best practice to start with data at only one or two sites. By construction, the grid should let you have data "on the grid" and not worry about where it is, but in practice you may be supported by site administrators, particularly in the beginning, and it may be best to start with one or two sites. (If you are a local VO, i.e. local to a site, that site would of course be your main resource and your main support.)
Information for sites (and prospective sites)
We recommend that you join the storage meeting - see above.
Information for everybody else
Storage experts in GridPP have a lot of expertise with storage systems: Mostly, of course Grid storage, but also maintaining the underlying storage clusters, distributed filesystems, and data management for science in general.
There are several key bits of software used within GridPP to provide grid-enabled storage.