Oxford/NGSCE
OBJECTIVE
The purpose of this exercise was to set up an lcg-CE at the Oxford NGS1 site so LCG2 / EGEE3 grid jobs could be submitted to NGS Cluster through the Workload Management System (WMS). We have installed an lcg-CE (ngsce-test.oerc.ox.ac.uk) on a virtual machine at the Oxford e-Research Centre (OeRC4). Ngsce-test.oerc.ox.ac.uk is registered in GOCDB5 as an lcg-CE under the UKI-SOUTHGRID-OX-HEP site. It is using t2se01.physics.ox.ac.uk as its storage element and publishing to the information system through the site bdii t2bdii.physics.ox.ac.uk, both of which are part of UKI-SOUTHGRID-OX-HEP.
We have installed the WN (Worker Node) tar ball of the gLite6 software on a shared area which is accessible from all worker nodes of the NGS Cluster. All computing nodes of NGS cluster also share software area which is used by the different Virtual Organisations (VO’s) to install their software. CE (Computing Element) and Worker Node share home directories.
NGS Setup
NGS at OeRC is using the PbsPro7 batch system. All home directories are shared across the network and the PbsPro client toolkit is installed on a shared area.
CE installation
Repository Setting: The following repositories were set up for installation.
CA
lcg-CE
glite-BDII
glite-TORQUE_utils
Jpackege
Due to some inconsistency between jpackage and glite, it is recommended to install java and jdk packages manually before installing any of gLite components.
yum install java-1.6.0-sun-compat.i586
yum install jdk.i586
Host Certificate:
lcg-CE needs a host certificate. Convert .p12 file into hostcert.pem and hostkey.pem and copy it to /etc/grid-security/
openssl pkcs12 -in my_cert.p12 -clcerts -nokeys -out hostcert.pem
openssl pkcs12 -in my_cert.p12 -nodes -nocerts -out hostkey.pem
chmod 400 hostkey.pem
chmod 444 hostcert.pem
Pool Account
gLite requires separate pool accounts for every supported VO to which real users are mapped by the CE. It is good practice to have a large number (~200) of pool accounts per VO so every grid user is permanently mapped to a unique unix account. A few special accounts, like sgm and prd are also required for software management and production jobs for every supported VO.
In normal gLite setup, pool accounts are created through yaim8 and the required information like UID and GID is provided by users.conf and groups.conf file. More information about yaim and pool account is available at https://twiki.cern.ch/twiki/bin/view/LCG/YaimGuide400
In our setup, pool accounts and groups were created manually through NIS (Network Information Service) and all the pool accounts are sharing home directories.
A few specific accounts are also required on the CE only. These were created manually on the CE only
adduser -d /lhome/edginfo edginfo
adduser -d /lhome/edguser edguser
adduser –d /lhome/glite glite.
Although users.conf and groups.conf files were not used for pool account creation yaim does however, require the presence of them for the configuration of the CE. A small script was written to create users.conf and groups.conf for pool accounts.
make_users users.conf groups.conf
The script is available here. http://www-pnp.physics.ox.ac.uk/~mohammad/make_users
Configuration of lcg-CE
lcg-CE is configured through YAIM. It is a script which runs different functions to configure the CE and takes all input parameters from the site-info.def file. It also requires users.conf and groups.conf in the correct format. Further information about yaim is here
https://twiki.cern.ch/twiki/bin/view/LCG/YaimGuide400
It is good practice to create a separate directory for configuration input files and keep all input files there. That configuration directory must also have a vo.d sub-directory which contains one file per VO, the name of which should be the VO name in lower case. These files contain definition of VOs from VO ID card, available on the CIC portal9.
Site-info.def :
Detailed information about mandatory and default variables for an lcg-CE is explained here.
https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#lcg_CE
Important variable specific to ngsce-test ce are fallowing.
CONFIG_USERS=no
GLITE_HOME_DIR=/lhome/glite
EDGINFO_HOME_DIR=/lhome/edginfo
EDG_HOME_DIR=/lhome/edguser
MY_DOMAIN=oerc.ox.ac.uk
CE_HOST=ngsce-test.$MY_DOMAIN
MON_HOST=t2mon02.physics.ox.ac.uk
SITE_BDII_HOST=t2bdii02.physics.ox.ac.uk
CONFIG_DIR=/home/kashif/yaim-config-test
USERS_CONF=${CONFIG_DIR}/users.conf
GROUPS_CONF=${CONFIG_DIR}/groups.conf
GLOBUS_TCP_PORT_RANGE="64000,65256"
BATCH_SERVER=10.141.255.254
BATCH_VERSION=PBSPro_8.0.0
BATCH_LOG_DIR=/var/spool/pbs/server_priv/accounting
BATCH_BIN_DIR=/usr/local/Cluster-Apps/PBSPro_8.0.0/bin/
JOB_MANAGER=lcgpbs
DPM_HOST="t2se01.physics.ox.ac.uk"
SE_LIST="$DPM_HOST"
SE_ACCESSPOINT=/dpm/physics.ox.ac.uk/home
VOS="cdf dteam dzero hone ilc mice ops pheno vo.southgrid.ac.uk ngs.ac.uk zeus"
WORKQ_GROUP_ENABLE="cdf dteam dzero hone ilc mice ops pheno vo.southgrid.ac.uk ngs.ac.uk zeus "
QUEUES="workq"
VO_SW_DIR=/usr/local/Cluster-Apps/glite/software
CLASSIC_HOST="t2se01.physics.ox.ac.uk"
CLASSIC_STORAGE_DIR="/dpm/physics.ox.ac.uk/home"
Complete site-info.def file is available here. http://www-pnp.physics.ox.ac.uk/~mohammad/site-info.def
Configuring the CE through YAIM
/opt/glite/yaim/bin/yaim -c -s site-info.def -n lcg-CE -n TORQUE_utils
Installation and Configuration of Tarball WN
WN tarball distribution is suitable for situations where it is not possible to install WN software on worker nodes. This distribution consists of two tarballs, one containing the glite software and the other containing external dependencies which are usually not installed on the host. The software should be accessible by all worker nodes.
Download the latest tarball distribution and un tar it http://grid-deployment.web.cern.ch/grid-deployment/download/relocatable/glite-WN/SL4_i686/
# tar -zxvf glite-WN-3.1.37-0-external.tar.gz
# tar -zxvf glite-WN-3.1.37-0.tar.gz
Creating a symlink to prod folder makes it easy to switch between versions. So in case of upgrade just point to the new version.
The configuration steps for a WN are similar to the CE, it just needs a few different variables.
INSTALL_ROOT=/usr/local/Cluster-Apps/glite/prod # where tarball is installed
CONFIG_DIR=/usr/local/Cluster-Apps/glite/yaim-config/site-info.def # for yaim config files
GLITE_EXTERNAL_ROOT=${INSTALL_ROOT}/external
GRID_ENV_LOCATION=${INSTALL_ROOT}/external/etc/profile.d
EDG_WL_SCRATCH=/scratch
CRON_DIR=${CONFIG_DIR}/cron
X509_CERT_DIR=${GLITE_EXTERNAL_ROOT}/etc/grid-security/certificates
X509_VOMS_DIR=${GLITE_EXTERNAL_ROOT}/etc/grid-security/vomsdir
#Rest same as CE
Configure the WN tarball
$INSTALL_ROOT/glite/yaim/bin/yaim -c –s site-info.def -n WN_TAR
To install CA certificate through yaim
$INSTALL_ROOT/glite/yaim/bin/yaim -r –s site-info.def -n WN_TAR -f config_certs_userland
It will install CA certificates in folder pointed by X509_CERT_DIR folder
Post Configuration Steps
Setting of pool account environment:
When a grid job arrives at a worker node it should load environment variables specific to grid jobs. All grid environment variables are specified in
${GLITE_EXTERNAL_ROOT}/etc/profile.d directory.
First add the following lines to ${GLITE_EXTERNAL_ROOT}/etc/profile.d/grid-env.sh file
gridenv_set "X509_VOMS_DIR" "${GLITE_EXTERNAL_ROOT} /grid-security/vomsdir"
gridenv_set "X509_CERT_DIR" “${GLITE_EXTERNAL_ROOT} /grid-security/certificates"
Second step is to make sure that every pool account has a .profile file with this content, so before starting any job they will have run the following lines.
.${GLITE_EXTERNAL_ROOT/etc/profile.d/a1_grid_env.sh
.${GLITE_EXTERNAL_ROOT/etc/profile.d/grid-env.sh
I have a small script which copies a .profile file to every grid pool account. It is here http://www-pnp.physics.ox.ac.uk/~mohammad/create_profile
Setting up of cron jobs
Yaim automatically sets up many cron jobs to clean job directories, fetch crl etc. The tar ball wn installation however does not set up any cron jobs. I have created this cron job to fetch crl (Certificate Revocation Lists) for WN on CE
PATH=/sbin:/bin:/usr/sbin:/usr/bin
8 20,2,8,14 * * * root ${GLITE_EXTERNAL_ROOT/usr/sbin/fetch-crl --loc ${GLITE_EXTERNAL_ROOT /etc/grid-security/certificates --out ${GLITE_EXTERNAL_ROOT/etc/grid- security/certificates -no-check-certificate >> /var/log/fetch-crl-cron-wn.log 2>&1
Other option is to set up a cron job which copies crl from CE to WN area.
Software Area
Software managers of VOs with valid VOMS role can install software for their respective VOs in the Software area. Software should be accessible by all WNs. It should be declared in vo.d directory as SW_DIR. Different software directories are created by YAIM.
Information System
Glite Information system has a hierarchal structure. It uses Berkley Database Information Index (BDII) as its building block . Information propagates from resource level BDII to top level BDII through site level BDII. We are using t2bdii02.physics.ox.ac.uk as site level BDII. In lcg-CE, resource level BDII is co-located with CE and it is populated by running plug-ins which in turn collects static and dynamic information by parsing the log and through batch system level commands.
PbsPro is not officially supported by glite so we have to change information plug-ins to support PbsPro. The output of the qstat and pbsnodes commands are different for PbsPro in comparison to the original PBS or torque10 batch system.
The original rpm was provided by Jan Just Keijser from Nikhef, Amsterdam . It has been changed to support new variables provided in latest information plugins. . The final version is available here. http://www-pnp.physics.ox.ac.uk/~mohammad/PbsPro.tgz
It comprises of four files
opt/lcg/libexec/lcg-info-dynamic-pbspro
opt/lcg/libexec/lrmsinfo-pbspro
opt/lcg/lib/python/PbsProServer.py
opt/lcg/lib/python/PbsPro_utils.py
And to use this, we have to edit these two files
/opt/glite/etc/gip/plugin/glite-info-dynamic-ce
/opt/glite/etc/lcg-info-dynamic-scheduler.conf
Accounting
EGEE gLite infrastructure uses Accounting Processor for Event Logs (APEL)11 component to collect and aggregate CPU usage information from sites. APEL parses batch system and Grid gatekeeper logs at CE to generate CPU usage records and send them to site glite-MON node which in turn publish them to a centralized repository using R-GMA12 as the transport mechanism.
Apel service running on CE needs access to pbs log file. We have set up a cron job at PBS head node which r-sync log files to ngsce-test. T2mon02.physics.ox.ac.uk is the glite-MON box running at Oxford EGEE site.
Problem and Issues
To be part of EGEE infrastructure, It is necessary that every node should pass SAM13 tests. If a node fails two consecutive critical tests, the SAM system raises an alarm which goes to the Operations Dashboard14 and consequently a ticket is raised against that site.
SAM jobs come with a limited proxy and it is the site responsibility that the job should be run and completed within 6 hours of arrival at site. Most of the site’s are using the Maui batch scheduler15 to make sure that ops and dteam jobs have highest priority or a job slot reserved for monitoring jobs. An other issue is that most of the grid jobs don’t use job pre-emption so if a job is pre-emptied by scheduler it wont be able to run properly again.
References
- http://www.ngs.ac.uk/
- LCG http://lcg.web.cern.ch/lcg/
- EGEE Enabling Grids for E-sciencE http://www.eu-egee.org/
- http://www.oerc.ox.ac.uk/
- Grid Operations Centre Database https://goc.gridops.org/
- http://glite.web.cern.ch/glite/
- http://www.pbsworks.com/
- http://yaim.info/
- http://cic.gridops.org/ The VO ID cards are available in the VO tab https://cic.gridops.org/index.php?section=vo
- http://www.clusterresources.com/products/torque-resource-manager.php
- http://www3.egee.cesga.es/gridsite/accounting/CESGA/egee_view.php
- http://www.r-gma.org
- Service Avilability Monitoring https://lcg-sam.cern.ch:8443/sam/sam.py
- https://operations-portal.in2p3.fr/ Operational Documentation is available on the CERN twiki https://twiki.cern.ch/twiki/bin/view/EGEE/OperationalDocumentation
- http://sourceforge.net/projects/mauischeduler
