Difference between revisions of "XRootD"
(Created page with "This instructions are an example of how to setup a multi-VO xrootd server that participates in a GridPP-wide storage network.<br> Right now (03/2017) it is only between server...") |
m (Suggested update for invalid link) |
||
(5 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
This instructions are an example of how to setup a multi-VO xrootd server that participates in a GridPP-wide storage network.<br> | This instructions are an example of how to setup a multi-VO xrootd server that participates in a GridPP-wide storage network.<br> | ||
Right now (03/2017) it is only between servers withing ScotGrid and in testing mode.<br> | Right now (03/2017) it is only between servers withing ScotGrid and in testing mode.<br> | ||
− | A presentation about different setup possibilities can be found [https://docs.google.com/presentation/d/1Doztqt66GbAowupGJAssegDQa0WqO5T3U_FFMwjBiNo here]<br> | + | A presentation about different setup possibilities can be found ---> [https://docs.google.com/presentation/d/1Doztqt66GbAowupGJAssegDQa0WqO5T3U_FFMwjBiNo here]<br> |
− | + | ||
− | + | ||
+ | general information about xrootd/cmsd are available ---> [http://xrootd.org/doc/prod/cms_config.htm here] (181114 404!) [[https://xrootd.slac.stanford.edu/doc/dev54/cms_config.htm maybe this page]]? | ||
==Instruction to setup a disk server for xrootd== | ==Instruction to setup a disk server for xrootd== | ||
Line 27: | Line 26: | ||
** cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem | ** cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem | ||
** chown -R xrootd:xrootd /etc/grid-security/xrd | ** chown -R xrootd:xrootd /etc/grid-security/xrd | ||
− | * create a proper home directory for user xrootd | + | * create a proper home directory for user xrootd: /home/xrootd |
** change ownership of it to xrootd:xrootd | ** change ownership of it to xrootd:xrootd | ||
** enable it as xrootd's homedirectory in /etc/passwd | ** enable it as xrootd's homedirectory in /etc/passwd | ||
Line 36: | Line 35: | ||
** use /etc/shadow to disable logins for xrootd (preferred and probably default) | ** use /etc/shadow to disable logins for xrootd (preferred and probably default) | ||
* create cronjob for root to execute edg-mkgridmap | * create cronjob for root to execute edg-mkgridmap | ||
− | ** 10 */6 * * * /usr/sbin/edg-mkgridmap --conf= | + | ** 10 */6 * * * /usr/sbin/edg-mkgridmap --conf=/home/xrootd/config/edg-mkgridmap.conf --output=/etc/grid-security/grid-mapfile --safe |
* create as user xrootd needed directories | * create as user xrootd needed directories | ||
** $HOME/config | ** $HOME/config | ||
Line 52: | Line 51: | ||
** xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/check-running.sh $HOME/bin/check-running.sh | ** xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/check-running.sh $HOME/bin/check-running.sh | ||
* make all the downloaded scripts executable | * make all the downloaded scripts executable | ||
− | ** chmod +x | + | ** chmod +x /home/xrootd/bin/* |
* as user xrootd create a $HOME/local.cfg with the content: export localstorage=/PATH/To/Your/Grid/Storage/Space | * as user xrootd create a $HOME/local.cfg with the content: export localstorage=/PATH/To/Your/Grid/Storage/Space | ||
** it should point to the directory where you have mounted the space that xrootd can use | ** it should point to the directory where you have mounted the space that xrootd can use | ||
** e.g.: export localstorage=/gridstorage | ** e.g.: export localstorage=/gridstorage | ||
* make sure your grid storage space is owned and writeable by user&group xrootd | * make sure your grid storage space is owned and writeable by user&group xrootd | ||
− | ** as root: source | + | ** as root: source /home/xrootd/local.cfg && chown -R xrootd:xrootd $localstorage |
* as root execute edg-mkgridmap once | * as root execute edg-mkgridmap once | ||
− | ** edg-mkgridmap --conf= | + | ** edg-mkgridmap --conf=/home/xrootd/config/edg-mkgridmap.conf --output=/etc/grid-security/grid-mapfile --safe |
* as root disable xrootd as a system service | * as root disable xrootd as a system service | ||
** chkconfig xrootd off | ** chkconfig xrootd off | ||
Line 72: | Line 71: | ||
** exit | ** exit | ||
− | To check if your server is registered with the redirector: xrdfs xroot://dev2.gridpp.ecdf.ed.ac.uk locate -m \* <br> | + | |
− | To (re)start the service, do as user xrootd: $HOME/bin/start-xrootd-server <br> | + | We have 2 redirectors in the cluster through which all servers can be reached and which also work in a load balanced way: |
− | To end the service, do as user xrootd: $HOME/bin/end-xrootd-server <br> | + | dev2.gridpp.ecdf.ed.ac.uk |
+ | dpm-dev.gla.scotgrid.ac.uk | ||
+ | |||
+ | To check if your server is registered with the redirector: <br> | ||
+ | ''xrdfs xroot://dev2.gridpp.ecdf.ed.ac.uk locate -m \* '' <br> | ||
+ | or: ''xrdfs xroot://dpm-dev.gla.scotgrid.ac.uk locate -m \* ''<br><br> | ||
+ | To (re)start the service, do as user xrootd: ''$HOME/bin/start-xrootd-server ''<br> | ||
+ | To end the service, do as user xrootd: ''$HOME/bin/end-xrootd-server ''<br> | ||
+ | |||
+ | To transfer data to the cluster: <br> | ||
+ | ''xrdcp xroot://REDIRECTOR//VO/USERNAME/FILE ''<br> | ||
+ | where<br> | ||
+ | REDIRECTOR: one of the 2 redirectors mentioned above<br> | ||
+ | VO: the VO name you have a voms proxy for (gridpp, atlas, dteam, lhcb, lsst), other VOs can easily be enabled<br> | ||
+ | USERNAME: your username<br> | ||
+ | FILE: the file you want to transfer | ||
+ | |||
+ | Read access is enabled already at the VO level and users should only be able to remove own files. |
Latest revision as of 13:52, 12 June 2023
This instructions are an example of how to setup a multi-VO xrootd server that participates in a GridPP-wide storage network.
Right now (03/2017) it is only between servers withing ScotGrid and in testing mode.
A presentation about different setup possibilities can be found ---> here
general information about xrootd/cmsd are available ---> here (181114 404!) [maybe this page]?
Instruction to setup a disk server for xrootd
- do a basic/minimal OS install (SL6 tested, CentOS7 used in Glasgow too)
- open ports 1094 and 3121
- install general GridPP infrastructure
- hostkey/cert in /etc/grid-security with correct permissions
- install edg-mkgridmap
- install certificates: ca_*
- install the gridpp-voms package (all)
- create unix groups for all VOs
- list of all VOs is given by: ls -1 /etc/grid-security/vomsdir
- One of the main principles of this storage is that all sites support all VOs for storage, there is no more coupling between what is supported for computing on a site and the local storage. However, quotas can be used at the file system level (e.g. through ZFS) to restrict usage for specific VOs.
- install packages needed for xrootd to work
- xrootd
- vomsxrd
- xrootd-client
- this list will be updated once http and gridftp transfers through xrootd are supported (and needed)
- copy hostkey/cert where xrootd expect it
- mkdir /etc/grid-security/xrd
- cp /etc/grid-security/hostkey.pem /etc/grid-security/xrd/xrdkey.pem
- cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem
- chown -R xrootd:xrootd /etc/grid-security/xrd
- create a proper home directory for user xrootd: /home/xrootd
- change ownership of it to xrootd:xrootd
- enable it as xrootd's homedirectory in /etc/passwd
- change shell for xrootd in /etc/passwd to /bin/bash
- make sure user xrootd can't login in remotely through ssh
- create a strong password for user xrootd, OR
- disable password based ssh logins, OR
- use /etc/shadow to disable logins for xrootd (preferred and probably default)
- create cronjob for root to execute edg-mkgridmap
- 10 */6 * * * /usr/sbin/edg-mkgridmap --conf=/home/xrootd/config/edg-mkgridmap.conf --output=/etc/grid-security/grid-mapfile --safe
- create as user xrootd needed directories
- $HOME/config
- $HOME/bin
- $HOME/log
- $HOME/spool
- $HOME/run
- as user xrootd copy needed config and administrating files to your server
- xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/xrootd-gridpp.cfg $HOME/config/xrootd-gridpp.cfg
- xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/auth_file $HOME/config/auth_file
- xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/edg-mkgridmap.conf $HOME/config/edg-mkgridmap.conf
- xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/check-config-gridpp.sh $HOME/bin/check-config-gridpp.sh
- xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/start-xrootd-server.sh $HOME/bin/start-xrootd-server.sh
- xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/end-xrootd-server.sh $HOME/bin/end-xrootd-server.sh
- xrdcp -f xroot://gridpp09.ecdf.ed.ac.uk//etc/check-running.sh $HOME/bin/check-running.sh
- make all the downloaded scripts executable
- chmod +x /home/xrootd/bin/*
- as user xrootd create a $HOME/local.cfg with the content: export localstorage=/PATH/To/Your/Grid/Storage/Space
- it should point to the directory where you have mounted the space that xrootd can use
- e.g.: export localstorage=/gridstorage
- make sure your grid storage space is owned and writeable by user&group xrootd
- as root: source /home/xrootd/local.cfg && chown -R xrootd:xrootd $localstorage
- as root execute edg-mkgridmap once
- edg-mkgridmap --conf=/home/xrootd/config/edg-mkgridmap.conf --output=/etc/grid-security/grid-mapfile --safe
- as root disable xrootd as a system service
- chkconfig xrootd off
- different for CentOS7
- as user xrootd create a cronjob
- */30 * * * * $HOME/bin/check-running.sh
- it checks (every 30min here) if all services are running and restarts everything if needed
- also makes sure after a power outage or reboot the service comes up by its own after a while
- as user xrootd start the service
- su - xrootd
- $HOME/bin/start-xrootd-server.sh
- exit
We have 2 redirectors in the cluster through which all servers can be reached and which also work in a load balanced way:
dev2.gridpp.ecdf.ed.ac.uk
dpm-dev.gla.scotgrid.ac.uk
To check if your server is registered with the redirector:
xrdfs xroot://dev2.gridpp.ecdf.ed.ac.uk locate -m \*
or: xrdfs xroot://dpm-dev.gla.scotgrid.ac.uk locate -m \*
To (re)start the service, do as user xrootd: $HOME/bin/start-xrootd-server
To end the service, do as user xrootd: $HOME/bin/end-xrootd-server
To transfer data to the cluster:
xrdcp xroot://REDIRECTOR//VO/USERNAME/FILE
where
REDIRECTOR: one of the 2 redirectors mentioned above
VO: the VO name you have a voms proxy for (gridpp, atlas, dteam, lhcb, lsst), other VOs can easily be enabled
USERNAME: your username
FILE: the file you want to transfer
Read access is enabled already at the VO level and users should only be able to remove own files.