WLCG SRM usage

From GridPP Wiki
Jump to: navigation, search

The script below queries the WLCG top-level BDII for information about the deployed storage elements at participating WLCG sites. The GLUE schema has an entry GlueSARoot which (in the default SRM configurations) contains information about the SRM implementation that is being used at a site.

#! /bin/bash
 
DCACHE=/tmp/wlcg-dcache.txt
DPM=/tmp/wlcg-dpm.txt
CASTOR=/tmp/wlcg-castor.txt
 
#ldapsearch -x -h lcg-bdii.cern.ch:2170 -b o=grid | grep '^GlueSAPath.*/pnfs/' | sed 's-.*/pnfs/--;s-/.*--' | sort -u > $DCACHE
#ldapsearch -x -h lcg-bdii.cern.ch:2170 -b o=grid | grep '^GlueSAPath.*/dpm/' | sed 's-.*/dpm/--;s-/.*--' | sort -u > $DPM
#ldapsearch -x -h lcg-bdii.cern.ch:2170 -b o=grid | grep '^GlueSAPath.*/castor/' | sed 's-.*/castor/--;s-/.*--' | sort -u > $CASTOR
 
dcacheTot=0
dpmTot=0
castorTot=0
 
echo TLD $'\t dCache\t DPM\t CASTOR\t Country Total'
 
for country in `grep -o -h -P "(\.[\w]{2,3})$" $DCACHE $DPM $CASTOR|sort -u`
do
  dcache=`grep -P "$country$" $DCACHE |wc -l`
  dpm=`grep -P "$country$" $DPM |wc -l`
  castor=`grep -P "$country$" $CASTOR |wc -l`
 
  countryTot=$[dcache+dpm+castor]
 
  dcacheTot=$[dcacheTot+dcache]
  dpmTot=$[dpmTot+dpm]
  castorTot=$[castorTot+castor]
 
  echo $country $'\t' $dcache $'\t' $dpm $'\t' $castor $'\t' $countryTot
 
done
total=$[dcacheTot+dpmTot+castorTot]
echo
echo "totals" $'\t' $dcacheTot $'\t' $dpmTot $'\t' $castorTot $'\t' $total


15/12/2006

As of 15th Dec 2006, the distribution of the dCache, DPM and CASTOR SRMs was as follows (where TLD is the top level domain of the country):

TLD      dCache  DPM     CASTOR  Country Total
.at      0       1       0       1
.au      0       1       0       1
.be      1       1       0       2
.bg      1       1       0       2
.br      1       0       0       1
.ca      2       1       0       3
.ch      0       1       1       2
.cn      1       0       0       1
.com     0       1       0       1
.cz      0       1       0       1
.de      9       1       0       10
.edu     2       0       0       2
.ee      1       0       0       1
.es      1       2       3       6
.fr      1       4       0       5
.gov     1       0       0       1
.gr      1       4       0       5
.hr      0       3       0       3
.hu      0       4       0       4
.il      0       1       0       1
.in      0       1       0       1
.it      2       8       1       11
.jp      0       2       0       2
.kr      1       0       0       1
.lv      0       1       0       1
.nl      1       1       0       2
.org     1       1       0       2
.pk      0       1       0       1
.pl      0       3       0       3
.pt      0       1       0       1
.ru      2       7       0       9
.su      0       1       0       1
.tw      0       6       1       7
.ua      0       1       0       1
.uk      6       13      1       20
 
totals   35      74      7       116

Some conclusions can be drawn from this list.

  • Belgium, Bulgaria, Canada, Spain, France, Greece, Italy, Holland, .org, Russia and the UK all use dCache and DPM.
  • Italy, Spain and the UK support all 3 SRMs.
  • The UK has the largest number of deployed SRMs (21).

16/11/2007

TLD      dCache  DPM     CASTOR  Country Total
.am      0       1       0       1
.at      0       3       0       3
.au      0       2       0       2
.be      1       1       0       2
.bg      2       0       0       2
.br      1       1       0       2
.ca      4       1       0       5
.ch      1       1       1       3
.cn      1       1       0       2
.com     0       1       0       1
.cy      0       1       0       1
.cz      0       2       0       2
.de      10      1       0       11
.dk      1       0       0       1
.edu     5       0       0       5
.es      3       3       3       9
.fr      1       6       0       7
.gov     2       0       0       2
.gr      1       4       0       5
.hk      0       1       0       1
.hr      0       3       0       3
.hu      0       4       0       4
.ie      0       4       0       4
.il      0       1       0       1
.in      0       1       0       1
.int     0       1       0       1
.it      3       10      1       14
.jp      0       2       0       2
.kr      1       1       0       2
.lt      0       4       0       4
.lv      0       1       0       1
.mk      0       1       0       1
.nl      2       3       0       5
.org     2       1       0       3
.pk      0       1       0       1
.pl      0       4       0       4
.pt      1       4       0       5
.ro      0       2       0       2
.ru      3       7       0       10
.sg      0       1       0       1
.si      1       0       0       1
.sk      1       1       0       2
.su      0       1       0       1
.tr      0       6       0       6
.tw      0       7       1       8
.ua      0       1       0       1
.uk      7       13      1       21
.yu      0       1       0       1 
totals   54      116     7       177

Some conclusions can be drawn from this list.

  • There are 41 different countries supporting SRM interfaces to storage.
  • China, Italy, Spain and the UK support all 3 SRMs.
  • The UK has the largest number of deployed SRMs (21).

24th May 2010

Type Count World 
11
BeStMan 18
CASTOR 18
DPM 242
StoRM 45
bestman 7
dCache 72
dcache 5
Type Count UK
CASTOR 2 10
CASTOR 2.1.7-27 1
ALL CASTOR 11
DPM 1.7.2 6
DPM 1.7.2-4 6
DPM 1.7.2-5 1
DPM 1.7.3 4
DPM 1.7.3-1 3
ALL DPM 20
StoRM 1.3.19 1
StoRM 1.4.0 1
StoRM 1.5.1 3
ALL StoRM 5
dCache production-1.9.5-12 1
dCache production-1.9.5-15 1
ALL dCache 2

Table derived from Wahid's SRM Monitoring page, for the World. As can be seen, there's rather a lot of DPM out there.

Caveats

  • Not necessarily all of the SRMs listed in the global BDII will be in production.
  • Only six OSG sites appear on the list (.gov, .edu).
  • Only dCache, DPM and CASTOR are considered even though there are other systems in use. It is not clear at this time how to get information about these other systems from the data available through the top level BDII.