RAL Tier1 Workload Management WMS

From GridPP Wiki
Jump to: navigation, search

The RAL Tier1 runs a Workload Management System (WMS) and Logging and Bookeeping Subsystem (LB) on 3 glite-WMS and 2 glite-LB servers. Each WMS server uses an internal load balancing mechanism to access the both LB servers.

Service Endpoints

A list of VOs that the WMS servers support can be found from

  ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 \
      -b 'Mds-vo-name=RAL-LCG2,o=Grid' '(GlueServiceType=org.glite.wms.WMProxy)' \
      GlueServiceAccessControlRule

Similar for LB servers

  ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 \
      -b 'Mds-vo-name=RAL-LCG2,o=Grid' '(GlueServiceType=org.glite.lb.Server)' \
      GlueServiceAccessControlRule

As on 15 April 2009

WMS01 (lcgwms01.gridpp.rl.ac.uk) and WMS02 (lcgwms02.gridpp.rl.ac.uk) accept jobs only from LHC VOs (Alice, ATLAS, CMS and LHCb) + dteam and ops.

WMS03 (lcgwms03.gridpp.rl.ac.uk) accepts jobs from non-LHC VOs only (biomed, zeus, hone, cdf, dzero, babar, pheno, t2k, esr, ilc, magic, minos.vo.gridpp.ac.uk, mice, fusion, geant4, cedar, manmace, gridpp, ngs.ac.uk, camont, totalep, vo.southgrid.ac.uk, vo.northgrid.ac.uk, vo.scotgrid.ac.uk, supernemo.vo.eu-egee.org, na48 vo.nanocmos.ac.uk, vo.londongrid.ac.uk) + dteam and ops.

LB01 (lcglb01.gridpp.rl.ac.uk) and LB02 (lcglb02.gridpp.rl.ac.uk) are used by all three WMS above mentioned, and therefore are for general use.

Basic Usage

A user interface can be configured to use any of the WMS servers (in the example below 'lcgwms01' can be replaced with 'lcgwms02' or 'lcgwms03')

 # /opt/glite/etc/dteam/glite_wmsui.conf
 [
    NSAddresses  =  {"lcgwms01.gridpp.rl.ac.uk"};
    MyProxyServer  =  "lcgrbp01.gridpp.rl.ac.uk";
    VirtualOrganisation  =  "dteam";
    LBAddresses  =  {};
    HLRLocation  =  "";
 ]
 # /opt/glite/etc/dteam/glite_wms.conf
 [
    OutputStorage  =  "/tmp/jobOutput";
    JdlDefaultAttributes =  [
        RetryCount  =  3;
        rank  = - other.GlueCEStateEstimatedResponseTime;
        PerusalFileEnable  =  false;
        AllowZippedISB  =  true;
        requirements  =  other.GlueCEStateStatus == "Production";
        ShallowRetryCount  =  10;
        SignificantAttributes  =  {"Requirements", "Rank", "FuzzyRank"};
        MyProxyServer  =  "lcgrbp01.gridpp.rl.ac.uk";
        ];
    WMProxyServiceDiscoveryType  =  "org.glite.wms.wmproxy";
    virtualorganisation  =  "dteam";
    ErrorStorage  =  "/tmp";
    EnableServiceDiscovery  =  true;
    ListenerStorage  =  "/tmp";
    LBServiceDiscoveryType  =  "org.glite.lb.server";
    WMProxyEndpoints  =  {"https://lcgwms01.gridpp.rl.ac.uk:7443/glite_wms_wmproxy_server"};
 ]


Check the matching CEs for a JDL job (with glite-wms-job-match) and finally submit a job with glite-wms-job-submit


Service Monitoring


Local Deployment Information

The RAL Tier1 glite-WMS servers are deployed as it follows:

lcgwms01 and lcgwms02.gridpp.rl.ac.uk for LHC VO job submissions only

lcgwms03.gridpp.rl.ac.uk for non-LHC VO job submissions only

The LHC VO users have the possibility to load balance the two WMS servers by using some appropriate configuration when submitting jobs. At the UI level (or central job submission mechanism if any)

1. /opt/glite/etc/$VO/glite_wmsui.conf should contain

  NSAddresses  =  {"lcgwms01.gridpp.rl.ac.uk lcgwms02.gridpp.rl.ac.uk"};
  LBAddresses  =  {};

2. /opt/glite/etc/$VO/glite_wms.conf should contain

  WMProxyEndpoints  =  {"https://lcgwms01.gridpp.rl.ac.uk:7443/glite_wms_wmproxy_server",
                        "https://lcgwms02.gridpp.rl.ac.uk:7443/glite_wms_wmproxy_server"};

The theory (and the tests as well) says that the glite-wms-job-submit command will randomly pick a WMS from the list. If that WMS fails to accept the job, the next WMS will be tried, and so on. Once the job has been submitted successfully, it is tied to the WMS that accepted it.

The LB servers (lcglb01 and lcglb02) are accessed in a load balancing manner by all WMS servers using an internal configuration.

How to 'drain' a LB?

Assuming a LB has a RAID disk failure, it might be useful to stop WMSes sending jobs there to ease the load on it until the faulty disk is replaced. To do that, one can edit /opt/glite/etc/glite_wms.conf and remove the degraded LB server from the line:

  LBServer  =  {"lcglb01.gridpp.rl.ac.uk:9000","lcglb02.gridpp.rl.ac.uk:9000"};

in the WorkloadManagerProxy section.

Then restart the WMProxy to have it become effective immediately, otherwise existing WMProxy processes will keep using the old value until they exit after MaxServedRequests = 50.

Known Problems

Bulk submissions

Although one of the main reasons for glite-WMS development and deployment was the bulk job submissions, currently only bulk submissions of max 50 jobs are recommended. A known bug (click here) makes submissions of 50+ jobs to fail.


SandboxDir output file size

Currently no limit can be imposed on the size of job output files. Therefore users are recommended to carefully operate with output files, and also are kindly requested to retrieve these files (i.e. glite-wms-job-output) once the job is terminated.


No compatible resources

If a job fails and the WMS finds the job's "token" file still present (in the job's sandbox area), it means the job already exited before the WMS job wrapper was started. In that case the WMS can try a shallow resubmission (as allowed by the JDL), but that fails because of this bug. When a resubmission happens, previously used CEs are not considered at all, and so if there aren't "new" CEs the job is aborted because "not compatible resources were found".

Other Resources