RAL Tier1 Workload Management WMS
The RAL Tier1 runs a Workload Management System (WMS) and Logging and Bookeeping Subsystem (LB) on 3 glite-WMS and 2 glite-LB servers. Each WMS server uses an internal load balancing mechanism to access the both LB servers.
Contents
Service Endpoints
A list of VOs that the WMS servers support can be found from
ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 \ -b 'Mds-vo-name=RAL-LCG2,o=Grid' '(GlueServiceType=org.glite.wms.WMProxy)' \ GlueServiceAccessControlRule
Similar for LB servers
ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 \ -b 'Mds-vo-name=RAL-LCG2,o=Grid' '(GlueServiceType=org.glite.lb.Server)' \ GlueServiceAccessControlRule
As on 15 April 2009
WMS01 (lcgwms01.gridpp.rl.ac.uk) and WMS02 (lcgwms02.gridpp.rl.ac.uk) accept jobs only from LHC VOs (Alice, ATLAS, CMS and LHCb) + dteam and ops.
WMS03 (lcgwms03.gridpp.rl.ac.uk) accepts jobs from non-LHC VOs only (biomed, zeus, hone, cdf, dzero, babar, pheno, t2k, esr, ilc, magic, minos.vo.gridpp.ac.uk, mice, fusion, geant4, cedar, manmace, gridpp, ngs.ac.uk, camont, totalep, vo.southgrid.ac.uk, vo.northgrid.ac.uk, vo.scotgrid.ac.uk, supernemo.vo.eu-egee.org, na48 vo.nanocmos.ac.uk, vo.londongrid.ac.uk) + dteam and ops.
LB01 (lcglb01.gridpp.rl.ac.uk) and LB02 (lcglb02.gridpp.rl.ac.uk) are used by all three WMS above mentioned, and therefore are for general use.
Basic Usage
A user interface can be configured to use any of the WMS servers (in the example below 'lcgwms01' can be replaced with 'lcgwms02' or 'lcgwms03')
# /opt/glite/etc/dteam/glite_wmsui.conf [ NSAddresses = {"lcgwms01.gridpp.rl.ac.uk"}; MyProxyServer = "lcgrbp01.gridpp.rl.ac.uk"; VirtualOrganisation = "dteam"; LBAddresses = {}; HLRLocation = ""; ]
# /opt/glite/etc/dteam/glite_wms.conf [ OutputStorage = "/tmp/jobOutput"; JdlDefaultAttributes = [ RetryCount = 3; rank = - other.GlueCEStateEstimatedResponseTime; PerusalFileEnable = false; AllowZippedISB = true; requirements = other.GlueCEStateStatus == "Production"; ShallowRetryCount = 10; SignificantAttributes = {"Requirements", "Rank", "FuzzyRank"}; MyProxyServer = "lcgrbp01.gridpp.rl.ac.uk"; ]; WMProxyServiceDiscoveryType = "org.glite.wms.wmproxy"; virtualorganisation = "dteam"; ErrorStorage = "/tmp"; EnableServiceDiscovery = true; ListenerStorage = "/tmp"; LBServiceDiscoveryType = "org.glite.lb.server"; WMProxyEndpoints = {"https://lcgwms01.gridpp.rl.ac.uk:7443/glite_wms_wmproxy_server"}; ]
Check the matching CEs for a JDL job (with glite-wms-job-match) and finally submit a job with glite-wms-job-submit
Service Monitoring
- Ganglia Host Level Monitoring lcgwms01
- Ganglia Host Level Monitoring lcgwms02
- Ganglia Host Level Monitoring lcgwms03
- Ganglia Host Level Monitoring lcglb01
- Ganglia Host Level Monitoring lcglb02
- WMS Metrics
Local Deployment Information
The RAL Tier1 glite-WMS servers are deployed as it follows:
lcgwms01 and lcgwms02.gridpp.rl.ac.uk for LHC VO job submissions only
lcgwms03.gridpp.rl.ac.uk for non-LHC VO job submissions only
The LHC VO users have the possibility to load balance the two WMS servers by using some appropriate configuration when submitting jobs. At the UI level (or central job submission mechanism if any)
1. /opt/glite/etc/$VO/glite_wmsui.conf should contain
NSAddresses = {"lcgwms01.gridpp.rl.ac.uk lcgwms02.gridpp.rl.ac.uk"}; LBAddresses = {};
2. /opt/glite/etc/$VO/glite_wms.conf should contain
WMProxyEndpoints = {"https://lcgwms01.gridpp.rl.ac.uk:7443/glite_wms_wmproxy_server", "https://lcgwms02.gridpp.rl.ac.uk:7443/glite_wms_wmproxy_server"};
The theory (and the tests as well) says that the glite-wms-job-submit command will randomly pick a WMS from the list. If that WMS fails to accept the job, the next WMS will be tried, and so on. Once the job has been submitted successfully, it is tied to the WMS that accepted it.
The LB servers (lcglb01 and lcglb02) are accessed in a load balancing manner by all WMS servers using an internal configuration.
How to 'drain' a LB?
Assuming a LB has a RAID disk failure, it might be useful to stop WMSes sending jobs there to ease the load on it until the faulty disk is replaced. To do that, one can edit /opt/glite/etc/glite_wms.conf and remove the degraded LB server from the line:
LBServer = {"lcglb01.gridpp.rl.ac.uk:9000","lcglb02.gridpp.rl.ac.uk:9000"};
in the WorkloadManagerProxy section.
Then restart the WMProxy to have it become effective immediately, otherwise existing WMProxy processes will keep using the old value until they exit after MaxServedRequests = 50.
Known Problems
Bulk submissions
Although one of the main reasons for glite-WMS development and deployment was the bulk job submissions, currently only bulk submissions of max 50 jobs are recommended. A known bug (click here) makes submissions of 50+ jobs to fail.
SandboxDir output file size
Currently no limit can be imposed on the size of job output files. Therefore users are recommended to carefully operate with output files, and also are kindly requested to retrieve these files (i.e. glite-wms-job-output) once the job is terminated.
No compatible resources
If a job fails and the WMS finds the job's "token" file still present (in the job's sandbox area), it means the job already exited before the WMS job wrapper was started. In that case the WMS can try a shallow resubmission (as allowed by the JDL), but that fails because of this bug. When a resubmission happens, previously used CEs are not considered at all, and so if there aren't "new" CEs the job is aborted because "not compatible resources were found".
Other Resources
- The Workload Management Subsystem (WMS)
- The Logging and Bookeeping Subsystem (LB)
- glite WMS
- EGEE - Logging and Bookkeeping
- LB server quick Installation Guide
- Specific configuration on gLite WMS and LB 3.1 nodes for each VO
- Logging and Bookkeeping Documentation
- WMS User Guide
- WMS Troubleshooting Guide