From GridPP Wiki
Jump to: navigation, search

Oxford Links

Oxford Physics Department

Oxford Ganglia Statistics





CREAM CE with a Seperate batch system

yum install xml-commons-apis   due to a dependency problem
yum install lcg-CA glite-CREAM
yum install glite-TORQUE_server glite-TORQUE_utils


site-info.def : Cream specific attributes


Create a services directory in /etc/yaim (or anywhere you keep your site-info.def file) and copy glite-creamce from /opt/glite/yaim/examples/siteinfo/services/glite-creamce and edit it.

BLPARSER_HOST=t2ce02.physics.ox.ac.uk          # if torque server is same as creamce.

We are sharing gridmapdir for whole site

   t2torque02:/etc/grid-security/gridmapdir /etc/grid-security/gridmapdir    nfs udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0
   t2torque02:/opt/edg/var/info /opt/edg/var/info    nfs     udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0
   t2torque02:/opt/glite/var/info /opt/glite/var/info    nfs     udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0

Yaim complains about config_gip_vo_tag so created a blank function in /opt/glite/yaim/functions/local

And finaly run yaim

   /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n creamCE –n TORQUE_server -n TORQUE_utils

CREAM CE with shared batch cluster

The only notable difference is that since CREAM CE need pbs log file so it should be shared with cream ce

/etc/exports file at torque server

    /var/torque/server_priv/accounting t2ce04.physics.ox.ac.uk(ro,no_root_squash,async)

/etc/fstab at creamce

   t2torque02:/var/torque/server_priv/accounting /var/torque/server_priv/accounting nfs udp,hard,intr,exec,rw 0 0


   yum install xml-commons-apis   due to a dependency problem
   yum install lcg-CA glite-CREAM
   yum install glite-TORQUE_utils


   /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n creamCE -n TORQUE_utils

SCAS Server

Installed it on a SL5 machine and it requires host certificate


Dowload scas repo. http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.1/glite-SCAS.repo

   yum install lcg-CA glite-SCAS

SCAS server require only few general mandatory yaim variables and no service specific variable.

   * VO_<vo-name>_VOMSES
   * VO_<vo-name>_VOMS_CA_DN
   * VO_<vo-name>_VOMS_SERVERS
   * VOS

Create users.conf, groups.conf and vo.d directory as usual. It is also sharing gridmapdir

  t2torque02:/etc/grid-security/gridmapdir /etc/grid-security/gridmapdir  nfs udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0

Finally run yaim

   /opt/glite/yaim/bin/yaim -c -s site-info.def -n SCAS

Post Configuration Issues

fetch-crl is not enabled by default

 service fetch-crl-cron start

gridmapdir should be owned by scas. Since we are sharing gridmapdir between many CE,s so runing yaim changes the ownership of gridmapdir on scas server. In CE, it does not matter as it rums all the command as root so we edited yaim function to not change the ownership of gridmapdir. In creamce, edited /opt/glite/yaim/functions/config_vomsmap

     #chown root:${EDG_GROUP} ${GRIDMAPDIR}

In lcgce it is neccesary to edit config_mkgridmap as well

In our case default lcmaps-scas.db did not work with us so I have changed the order in which lcmaps calls its plugins and it worked!

 vomslocalgroup -> vomslocalaccount
 vomslocalaccount -> good | vomspoolaccount
 poolaccount -> good
 localaccount -> good

Configuring CREAM to work with SCAS

yaim does not configure cream ce to work with SCAS server. so Install glite-security-lcmaps-plugins-scas-client-0.2.8-3.sl5.x86_64.rpm and glite-security-saml2-xacml2-c-lib-0.0.15-5.sl5.x86_64.rpm manually from preferabaly etics repository http://etics-repository.cern.ch:8080/repository/download/registered/org.glite/org.glite.security.lcmaps-plugins-scas-client/0.2.8/sl5_x86_64_gcc412

and http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GLEXEC_wn/sl5/x86_64/RPMS.release/.

copy hostcert.pem and hostkey.pem to tomcathostcert.pem and tomcatkey.pem

   -rw-r--r--   1 tomcat tomcat   2196 Dec 18 16:19 tomcathostcert.pem
   -r--------   1 tomcat tomcat   1891 Dec 18 16:19 tomcathostkey.pem

Edit /opt/glite/etc/lcmaps/lcmaps-glexec.db so lcmaps at CE authenticate with remote SCAS server rather that local lcmaps plugins

 posix_enf = "lcmaps_posix_enf.mod"
 " -maxuid 1"
 " -maxpgid 1"
 " -maxsgid 32"
 localaccount = "lcmaps_localaccount.mod"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 poolaccount = "lcmaps_poolaccount.mod"
 " -override_inconsistency"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 " -gridmapdir /etc/grid-security/gridmapdir"
 vomslocalgroup = "lcmaps_voms_localgroup.mod" 
 " -groupmapfile /etc/grid-security/groupmapfile"
 " -mapmin 0"
 vomslocalaccount = "lcmaps_voms_localaccount.mod"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 " -use_voms_gid"
 vomspoolaccount = "lcmaps_voms_poolaccount.mod"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 " -gridmapdir /etc/grid-security/gridmapdir"
 " -do_not_use_secondary_gids"
 proxycheck = "lcmaps_verify_proxy.mod"
            "-certdir /etc/grid-security/certificates"
 # policies
 #proxycheck -> vomslocalgroup
 #vomslocalgroup -> vomslocalaccount
 #vomslocalaccount -> posix_enf | vomspoolaccount
 #vomspoolaccount -> posix_enf
 #proxycheck -> localaccount
 #localaccount -> posix_enf | poolaccount
 #poolaccount -> posix_enf
 scasclient = "lcmaps_scas_client.mod"
            " -capath /etc/grid-security/certificates"
            " -cert /etc/grid-security/tomcathostcert.pem"
            " -key  /etc/grid-security/tomcathostkey.pem"
            " -endpoint https://t2scas01.physics.ox.ac.uk:8443"
            " -resourcetype ce"
            " -actiontype execute-now"
 proxycheck -> scasclient

Test It

glite-ce-job-submit -a -r t2ce02.physics.ox.ac.uk:8443/cream-pbs-express test.jdl
log at CE : /var/log/messages
Jan 21 14:23:49 t2ce02 glexec[4050]:    lcmaps_plugin_scas_client-plugin_run(): User "/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad" with first FQAN "/dteam/Role=NULL/Capability=NULL" Permitted at endpoint "https://t2scas01.physics.ox.ac.uk:8443"

Argus Server


Reference point



Get Argus repos


Installing and Preliminary Configuration of Argus service is quite straight forward,


Configuring Argus Server

If yaim completes successfully, the next step is to load policies

check /opt/argus/pap/bin/pap-admin list-policies

    default (local):

It means that pap-standalone server is running perfectly

Now create a policy which will allow dteam and /ops/Role=pilot to use glexec from WN. Save a policy, say dteam_policy in a file. For glexec Argus policy should resemble authorization policy of CE,i.e it should be authorize for all roles

    resource "http://authz-interop.org/xacml/resource/resource-type/wn" {
      obligation "http://glite.org/xacml/obligation/local-environment-map" {}
       action "http://glite.org/xacml/action/execute" {
         rule permit { vo = dteam }
         rule permit {pfqan = "/atlas/Role=pilot" }
         rule permit {pfqan = "/atlas/Role=lcgadmin" }
         rule permit {pfqan = "/atlas/Role=production" }
         rule permit {pfqan = "/atlas" }
          rule permit {pfqan = "/ops/Role=pilot" }

Important thing to note is that resource and action should be same as define in lcmaps-glexec.db file in WN

Load policy through pap-admin and reloadpolicy at pdp

     /opt/argus/pap/bin/pap-admin apf dteam_policy
     /etc/init.d/pdp reloadpolicy
     /etc/init.d/pepd clearcache

Test glexec from WN as describe below in local test. In case of error, log level can be increased in /opt/argus/*/conf/logging.xml file. Error log files are at /opt/argus/*/logs/process.log

Glexec on WN

Installation Download glite-GLEXEC_wn repo from get http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/glite-GLEXEC_wn.repo

 yum install glite-GLEXEC_wn

probable error

 file /opt/globus/bin/xmlsec1-config from install of vdt_globus_essentials-VDT1.10.1x86_64_rhap_5-4.x86_64 conflicts with file from package vdt_globus_essentials-VDT1.10.1x86_rhap_5-3.i386


 rpm -e --nodeps vdt_globus_essentials-VDT1.10.1x86_rhap_5-3  vdt_globus_essentials-VDT1.10.1x86_64_rhap_5-3

And then again install glite-GLEXEC_wn

Configuration for scas

copy /opt/glite/yaim/examples/siteinfo/services/glite-glexec_wn to your config_dir and edit it


Configuration for Argus


No change required in site-info.def file.

   /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n WN -n TORQUE_client -n glite-GLEXEC_wn

After Configuration, there are two files of interest for Argus

/opt/glite/etc/lcmaps/lcmaps-glexec.db, It defines resourceid and actionid which is required for Argus server configuration

     pepc        = "lcmaps_c_pep.mod"
                 "--pep-daemon-endpoint-url https://t2argus02.physics.ox.ac.uk:8154/authz"
                 " -resourceid http://authz-interop.org/xacml/resource/resource-type/wn"
                 " -actionid http://glite.org/xacml/action/execute"
                 " -capath /etc/grid-security/certificates/"
                 " -pep-certificate-mode implicit"
                  verify_proxy -> pepc
                  pepc -> posix_enf

/opt/glite/etc/glexec.conf, it defines white listed account for glexec and log level also can be set here

Local Test

Copy your personal proxy from UI to WN and change ownership of proxy to any pilot account, then log into that account

   export GLEXEC_CLIENT_CERT=/tmp/test_proxy
   export GLEXEC_SOURCE_PROXY=/tmp/test_proxy
   export X509_USER_PROXY=/tmp/test_proxy
/opt/glite/sbin/glexec /usr/bin/id
uid=15176(dteam174) gid=2688(dteam)

A central Nagios is also testing glexec on WN using ops pilot role


It requires ops pilot pool account


Pakiti is a package monitoring tool developed at CERN. Pakiti 2 is the latest version but it is more suitable for security release monitoring and we didn't find useful for package monitoring so we are using Pakiti 1. Details are here http://pakiti.sourceforge.net/index2.html

It is a very light weight tool and can be installed on any shared web server.

Pakiti Server

   wget http://sourceforge.net/projects/pakiti/files/pakiti/1-0.1/pakiti-server-1-0.1.noarch.rpm/download
   wget http://sourceforge.net/projects/pakiti/files/pakiti/1-0.1/pakiti-client-1-0.1.noarch.rpm/download
   yum install httpd php mysql-server php-mysql mod_ssl 
   yum localinstall pakiti-server-* pakiti-client-*

Configure MySQL server

configure mysql server exactly as describe here http://pakiti.sourceforge.net/index2.html

Note : Don't use special character in password, there is bug in pakiti which some time create problem.

Configure Apache

I am not using https. Create a pakiti.conf file in /etc/httpd/conf.d/

   <VirtualHost _default_:80>
   ServerName pplxconfig.physics.ox.ac.uk:80
     Alias /pakiti "/var/lib/pakiti/www/"
       ErrorLog /var/log/httpd/pakiti-error
       CustomLog /var/log/httpd/pakiti-access common
       CustomLog /var/log/httpd/ssl_request_pakiti_log ssl_combined
       Alias /feed "/var/lib/pakiti/feed"
       <Directory "/var/lib/pakiti/feed">
        Options +FollowSymLinks
               AllowOverride None
               Order allow,deny
               Allow from all
               DirectoryIndex index.php
       <Directory "/var/lib/pakiti/www/pakiti">
               Options +FollowSymLinks
               AllowOverride None
               Order allow,deny
               Allow from all
               DirectoryIndex index.php

Configure Pakiti Server

edit /etc/pakiti/pakiti-server.conf and put dbname and passwd as provided in mysql configuration

Pakiti client

wget http://sourceforge.net/projects/pakiti/files/pakiti/1-0.1/pakiti-client-1-0.1.noarch.rpm/download

rpm -i pakiti-client-1-0.1.noarch.rpm

Edit /etc/pakiti/pakiti-client.conf and change the server_url

Pakiti-client also creates a cron job in cron.daily

run /usr/sbin/pakiti -v to check that client is working properly or not.

Other Oxford resources