Oxford

CREAM GLEXEC and SCAS

CREAM CE

Installation

CREAM CE with a Seperate batch system

yum install xml-commons-apis   due to a dependency problem
yum install lcg-CA glite-CREAM
yum install glite-TORQUE_server glite-TORQUE_utils

Configuration

site-info.def : Cream specific attributes

   JOB_MANAGER=pbs
   CE_BATCH_SYS=pbs

Create a services directory in /etc/yaim (or anywhere you keep your site-info.def file) and copy glite-creamce from /opt/glite/yaim/examples/siteinfo/services/glite-creamce and edit it.

CEMON_HOST=t2ce02.physics.ox.ac.uk
CREAM_DB_USER=<anything>
CREAM_DB_PASSWORD=******
BLPARSER_HOST=t2ce02.physics.ox.ac.uk          # if torque server is same as creamce.
CREAM_CE_STATE=Production

We are sharing gridmapdir for whole site

   t2torque02:/etc/grid-security/gridmapdir /etc/grid-security/gridmapdir    nfs udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0
   t2torque02:/opt/edg/var/info /opt/edg/var/info    nfs     udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0
   t2torque02:/opt/glite/var/info /opt/glite/var/info    nfs     udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0

Yaim complains about config_gip_vo_tag so created a blank function in /opt/glite/yaim/functions/local

And finaly run yaim

   /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n creamCE –n TORQUE_server -n TORQUE_utils

CREAM CE with shared batch cluster

The only notable difference is that since CREAM CE need pbs log file so it should be shared with cream ce

/etc/exports file at torque server

    /var/torque/server_priv/accounting t2ce04.physics.ox.ac.uk(ro,no_root_squash,async)
 

/etc/fstab at creamce

   t2torque02:/var/torque/server_priv/accounting /var/torque/server_priv/accounting nfs udp,hard,intr,exec,rw 0 0

Installation

   yum install xml-commons-apis   due to a dependency problem
   yum install lcg-CA glite-CREAM
   yum install glite-TORQUE_utils

Configurations

   /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n creamCE -n TORQUE_utils

SCAS Server

Installed it on a SL5 machine and it requires host certificate

Installation

Dowload scas repo. http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.1/glite-SCAS.repo

   yum install lcg-CA glite-SCAS
   

SCAS server require only few general mandatory yaim variables and no service specific variable.

   * GROUPS_CONF
   * USERS_CONF
   * VO_<vo-name>_VOMSES
   * VO_<vo-name>_VOMS_CA_DN
   * VO_<vo-name>_VOMS_SERVERS
   * VOS

Create users.conf, groups.conf and vo.d directory as usual. It is also sharing gridmapdir

  t2torque02:/etc/grid-security/gridmapdir /etc/grid-security/gridmapdir  nfs udp,rsize=8192,wsize=8192,hard,intr,exec,rw 0 0

Finally run yaim

   /opt/glite/yaim/bin/yaim -c -s site-info.def -n SCAS

Post Configuration Issues

fetch-crl is not enabled by default

 service fetch-crl-cron start

gridmapdir should be owned by scas. Since we are sharing gridmapdir between many CE,s so runing yaim changes the ownership of gridmapdir on scas server. In CE, it does not matter as it rums all the command as root so we edited yaim function to not change the ownership of gridmapdir. In creamce, edited /opt/glite/yaim/functions/config_vomsmap

else
     #chown root:${EDG_GROUP} ${GRIDMAPDIR}

In lcgce it is neccesary to edit config_mkgridmap as well

In our case default lcmaps-scas.db did not work with us so I have changed the order in which lcmaps calls its plugins and it worked!

 voms_mapping:
 vomslocalgroup -> vomslocalaccount
 vomslocalaccount -> good | vomspoolaccount
 classic_poolaccount:
 poolaccount -> good
 static_account_mapping:
 localaccount -> good

Configuring CREAM to work with SCAS

yaim does not configure cream ce to work with SCAS server. so Install glite-security-lcmaps-plugins-scas-client-0.2.8-3.sl5.x86_64.rpm and glite-security-saml2-xacml2-c-lib-0.0.15-5.sl5.x86_64.rpm manually from preferabaly etics repository http://etics-repository.cern.ch:8080/repository/download/registered/org.glite/org.glite.security.lcmaps-plugins-scas-client/0.2.8/sl5_x86_64_gcc412

and http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GLEXEC_wn/sl5/x86_64/RPMS.release/.

copy hostcert.pem and hostkey.pem to tomcathostcert.pem and tomcatkey.pem

   -rw-r--r--   1 tomcat tomcat   2196 Dec 18 16:19 tomcathostcert.pem
   -r--------   1 tomcat tomcat   1891 Dec 18 16:19 tomcathostkey.pem

Edit /opt/glite/etc/lcmaps/lcmaps-glexec.db so lcmaps at CE authenticate with remote SCAS server rather that local lcmaps plugins

 posix_enf = "lcmaps_posix_enf.mod"
 " -maxuid 1"
 " -maxpgid 1"
 " -maxsgid 32"
 localaccount = "lcmaps_localaccount.mod"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 poolaccount = "lcmaps_poolaccount.mod"
 " -override_inconsistency"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 " -gridmapdir /etc/grid-security/gridmapdir"
 vomslocalgroup = "lcmaps_voms_localgroup.mod" 
 " -groupmapfile /etc/grid-security/groupmapfile"
 " -mapmin 0"
 vomslocalaccount = "lcmaps_voms_localaccount.mod"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 " -use_voms_gid"
 vomspoolaccount = "lcmaps_voms_poolaccount.mod"
 " -gridmapfile /etc/grid-security/grid-mapfile"
 " -gridmapdir /etc/grid-security/gridmapdir"
 " -do_not_use_secondary_gids"
 proxycheck = "lcmaps_verify_proxy.mod"
            "-certdir /etc/grid-security/certificates"
 # policies
 #withvoms:
 #proxycheck -> vomslocalgroup
 #vomslocalgroup -> vomslocalaccount
 #vomslocalaccount -> posix_enf | vomspoolaccount
 #vomspoolaccount -> posix_enf
 #standard:
 #proxycheck -> localaccount
 #localaccount -> posix_enf | poolaccount
 #poolaccount -> posix_enf
 scasclient = "lcmaps_scas_client.mod"
            " -capath /etc/grid-security/certificates"
            " -cert /etc/grid-security/tomcathostcert.pem"
            " -key  /etc/grid-security/tomcathostkey.pem"
            " -endpoint https://t2scas01.physics.ox.ac.uk:8443"
            " -resourcetype ce"
            " -actiontype execute-now"
 glexec_get_account:
 proxycheck -> scasclient

Test It

glite-ce-job-submit -a -r t2ce02.physics.ox.ac.uk:8443/cream-pbs-express test.jdl
log at CE : /var/log/messages
Jan 21 14:23:49 t2ce02 glexec[4050]:    lcmaps_plugin_scas_client-plugin_run(): User "/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=kashif mohammad" with first FQAN "/dteam/Role=NULL/Capability=NULL" Permitted at endpoint "https://t2scas01.physics.ox.ac.uk:8443"

Argus Server

Installation

Reference point

http://glite.web.cern.ch/glite/packages/R3.2/sl5_x86_64/deployment/glite-ARGUS/glite-ARGUS.asp

https://twiki.cern.ch/twiki/bin/view/EGEE/AuthorizationFramework

Get Argus repos

http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/glite-ARGUS.repo

Installing and Preliminary Configuration of Argus service is quite straight forward,

https://twiki.cern.ch/twiki/bin/view/EGEE/AuthzQSYumYaimInstall

Configuring Argus Server

If yaim completes successfully, the next step is to load policies

check /opt/argus/pap/bin/pap-admin list-policies

    default (local):

It means that pap-standalone server is running perfectly

Now create a policy which will allow dteam and /ops/Role=pilot to use glexec from WN. Save a policy, say dteam_policy in a file. For glexec Argus policy should resemble authorization policy of CE,i.e it should be authorize for all roles

    resource "http://authz-interop.org/xacml/resource/resource-type/wn" {
      obligation "http://glite.org/xacml/obligation/local-environment-map" {}
       action "http://glite.org/xacml/action/execute" {
         rule permit { vo = dteam }
         rule permit {pfqan = "/atlas/Role=pilot" }
         rule permit {pfqan = "/atlas/Role=lcgadmin" }
         rule permit {pfqan = "/atlas/Role=production" }
         rule permit {pfqan = "/atlas" }
         
          rule permit {pfqan = "/ops/Role=pilot" }

                }
     }

Important thing to note is that resource and action should be same as define in lcmaps-glexec.db file in WN

Load policy through pap-admin and reloadpolicy at pdp

     /opt/argus/pap/bin/pap-admin apf dteam_policy
     /etc/init.d/pdp reloadpolicy
     /etc/init.d/pepd clearcache

Test glexec from WN as describe below in local test. In case of error, log level can be increased in /opt/argus/*/conf/logging.xml file. Error log files are at /opt/argus/*/logs/process.log

Glexec on WN

Installation Download glite-GLEXEC_wn repo from get http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/glite-GLEXEC_wn.repo

 yum install glite-GLEXEC_wn

probable error

 file /opt/globus/bin/xmlsec1-config from install of vdt_globus_essentials-VDT1.10.1x86_64_rhap_5-4.x86_64 conflicts with file from package vdt_globus_essentials-VDT1.10.1x86_rhap_5-3.i386

Workaround:

 rpm -e --nodeps vdt_globus_essentials-VDT1.10.1x86_rhap_5-3  vdt_globus_essentials-VDT1.10.1x86_64_rhap_5-3

And then again install glite-GLEXEC_wn

Configuration for scas

copy /opt/glite/yaim/examples/siteinfo/services/glite-glexec_wn to your config_dir and edit it

   GLEXEC_WN_SCAS_ENABLED=yes
   SCAS_HOST=t2scas01.physics.ox.ac.uk
   SCAS_PORT=8443
   GLEXEC_WN_OPMODE=setuid
   GLEXEC_WN_LOG_DESTINATION=file
   GLEXEC_WN_LOG_FILE=/var/log/glexec/glexec_log
    
    

Configuration for Argus

   GLEXEC_WN_ARGUS_ENABLED="yes"
   ARGUS_PEPD_ENDPOINTS="https://t2argus02.physics.ox.ac.uk:8154/authz"
   GLEXEC_WN_OPMODE="setuid"
   GLEXEC_WN_LOG_DESTINATION=file
   GLEXEC_WN_LOG_FILE=/var/log/glexec/glexec_log
   GLEXEC_WN_INPUT_LOCK=flock
   GLEXEC_WN_TARGET_LOCK=flock

No change required in site-info.def file.

   /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n WN -n TORQUE_client -n glite-GLEXEC_wn

After Configuration, there are two files of interest for Argus

/opt/glite/etc/lcmaps/lcmaps-glexec.db, It defines resourceid and actionid which is required for Argus server configuration

     pepc        = "lcmaps_c_pep.mod"
                 "--pep-daemon-endpoint-url https://t2argus02.physics.ox.ac.uk:8154/authz"
                 " -resourceid http://authz-interop.org/xacml/resource/resource-type/wn"
                 " -actionid http://glite.org/xacml/action/execute"
                 " -capath /etc/grid-security/certificates/"
                 " -pep-certificate-mode implicit"
     glexec_get_account:
                  verify_proxy -> pepc
                  pepc -> posix_enf

/opt/glite/etc/glexec.conf, it defines white listed account for glexec and log level also can be set here

Local Test

Copy your personal proxy from UI to WN and change ownership of proxy to any pilot account, then log into that account

   export GLEXEC_CLIENT_CERT=/tmp/test_proxy
   export GLEXEC_SOURCE_PROXY=/tmp/test_proxy
   export X509_USER_PROXY=/tmp/test_proxy
    

/opt/glite/sbin/glexec /usr/bin/id
uid=15176(dteam174) gid=2688(dteam)

A central Nagios is also testing glexec on WN using ops pilot role

 https://samnag023.cern.ch/nagios/

It requires ops pilot pool account

Pakiti

Pakiti is a package monitoring tool developed at CERN. Pakiti 2 is the latest version but it is more suitable for security release monitoring and we didn't find useful for package monitoring so we are using Pakiti 1. Details are here http://pakiti.sourceforge.net/index2.html

It is a very light weight tool and can be installed on any shared web server.

Pakiti Server

   wget http://sourceforge.net/projects/pakiti/files/pakiti/1-0.1/pakiti-server-1-0.1.noarch.rpm/download
   wget http://sourceforge.net/projects/pakiti/files/pakiti/1-0.1/pakiti-client-1-0.1.noarch.rpm/download
   yum install httpd php mysql-server php-mysql mod_ssl 
   yum localinstall pakiti-server-* pakiti-client-*

Configure MySQL server

configure mysql server exactly as describe here http://pakiti.sourceforge.net/index2.html

Note : Don't use special character in password, there is bug in pakiti which some time create problem.

Configure Apache

I am not using https. Create a pakiti.conf file in /etc/httpd/conf.d/

   <VirtualHost _default_:80>
   ServerName pplxconfig.physics.ox.ac.uk:80
     Alias /pakiti "/var/lib/pakiti/www/"
       ErrorLog /var/log/httpd/pakiti-error
       CustomLog /var/log/httpd/pakiti-access common
       CustomLog /var/log/httpd/ssl_request_pakiti_log ssl_combined
       Alias /feed "/var/lib/pakiti/feed"
       <Directory "/var/lib/pakiti/feed">
        Options +FollowSymLinks
               AllowOverride None
               Order allow,deny
               Allow from all
               DirectoryIndex index.php
       </Directory>
       <Directory "/var/lib/pakiti/www/pakiti">
               Options +FollowSymLinks
               AllowOverride None
               Order allow,deny
               Allow from all
               DirectoryIndex index.php
       </Directory>
    </VirtualHost>

Configure Pakiti Server

edit /etc/pakiti/pakiti-server.conf and put dbname and passwd as provided in mysql configuration

Pakiti client

wget http://sourceforge.net/projects/pakiti/files/pakiti/1-0.1/pakiti-client-1-0.1.noarch.rpm/download

rpm -i pakiti-client-1-0.1.noarch.rpm

Edit /etc/pakiti/pakiti-client.conf and change the server_url

Pakiti-client also creates a cron job in cron.daily

run /usr/sbin/pakiti -v to check that client is working properly or not.