Difference between revisions of "DPM upgrade at Oxford"

From GridPP Wiki
Jump to: navigation, search
 
(2 intermediate revisions by one user not shown)
Line 38: Line 38:
 
     override_options  => $override_options
 
     override_options  => $override_options
 
     }
 
     }
 +
 +
Run puppet locally to setup mysql
 +
 +
    puppet apply mysql.pp
 +
 +
Load old mysql backup database to new db
 +
 +
    mysql –u root –p < mysql-dump-final.sql
 +
 +
It is required to update tables as mysql version has changed
 +
 +
  mysql_upgrade --upgrade-system-tables
 +
 +
It is worth logging into DB and compare some data from old one
 +
 +
Install dpm puppet modules from puppet forge on the DPM head node
 +
 +
  puppet module install lcgdm-gridftp
 +
  puppet module install lcgdm-xrootd
 +
  puppet module install lcgdm-voms
 +
  puppet module install puppetlabs-mysql --version=3.4
 +
  puppet module install lcgdm-dmlite
 +
  puppet module install lcgdm-lcgdm
 +
  puppet module install CERNOps-bdii
 +
  puppet module install saz-memcached
 +
  puppet module install erwbgy-limits
 +
  puppet module install puppet-fetchcrl
 +
 +
Run puppet with final dpm manifests. Our dpm manifest is here https://gist.github.com/kashif74/499fc3678b82fbcab2229b8bbec9188e
 +
 +
  puppet apply dpm.pp
 +
 +
Ideally the above command should setup DPM Head node
 +
 +
I have also updated DPM pool nodes rpm packages to latest dpm realease but not the OS.
 +
dpm manifest for pool node is here https://gist.github.com/kashif74/3b1a941933bc3688c896fc965859c011
 +
 +
=== Post Configuration ===
 +
 +
==== DNS requests ====
 +
 +
Our IT services complained  that DPM Head node is making too many DNS request. Added all pool nodes in /etc/hosts file of headnode and headnode in /etc/hosts file of poolnodes
 +
 +
Installed nscd for local dns caching
 +
 +
==== xrootd ====
 +
 +
latest version of xrootd > 4.8.0. is not in UMD4 so it need to be pulled from EPEL (Feb 2018) . This might resolve in future.
 +
 +
==== Atlas and CMS name2name library ====
 +
 +
For CMS : https://twiki.cern.ch/twiki/bin/view/DPM/DPMComponents_Dpm-Xrootd#CMS_federation
 +
 +
For ATLAS : https://twiki.cern.ch/twiki/bin/view/DPM/DPMComponents_Dpm-Xrootd#ATLAS_federation. Atlas rpm is not available from Centos7 repo so install it from SL6 repo.

Latest revision as of 11:49, 13 March 2018

Objective

Upgrade and migrate DPM Head node from one SL6 physical server to another Centos7 server keeping the same hostname (t2se01 in our case)

Preparation

After the start of the GOCDB downtime, stopped all DPM related services on head node and pool nodes. Used netstat to confirm that there is no live connection to any pool node.

Run the final backup of mysql database on the old headnode

  /usr/bin/mysqldump --user=$mysql_user --password=$mysql_pw --opt --all-databases --single-transaction | gzip -c > mysql-dump-final.sql.gz

Changed the name of the old headnode from t2se01 to t2se01tmp in DNS and DHCP server

New DPM head node

Installed the new machine as t2se01.

Setup mysql (MariaDB) using local puppet

   cat mysql.pp
   $mysql_root_pass   = "****"
   $db_user           = "dpmmgr"
   $db_pass           = "****"
   $db_host           = "localhost"
   $override_options = {
   'mysqld' => {
   'max_connections'    => '1000',
   'query_cache_size'   => '256M',
   'query_cache_limit'  => '1MB',
   'innodb_flush_method' => 'O_DIRECT',
   'innodb_buffer_pool_size' => '1000000000',
      }
   }
   class{"mysql::server":
   service_enabled => true,
   root_password   => "${mysql_root_pass}",
   override_options  => $override_options
   }

Run puppet locally to setup mysql

   puppet apply mysql.pp 

Load old mysql backup database to new db

   mysql –u root –p < mysql-dump-final.sql

It is required to update tables as mysql version has changed

  mysql_upgrade --upgrade-system-tables

It is worth logging into DB and compare some data from old one

Install dpm puppet modules from puppet forge on the DPM head node

  puppet module install lcgdm-gridftp
  puppet module install lcgdm-xrootd
  puppet module install lcgdm-voms
  puppet module install puppetlabs-mysql --version=3.4
  puppet module install lcgdm-dmlite
  puppet module install lcgdm-lcgdm
  puppet module install CERNOps-bdii
  puppet module install saz-memcached
  puppet module install erwbgy-limits
  puppet module install puppet-fetchcrl

Run puppet with final dpm manifests. Our dpm manifest is here https://gist.github.com/kashif74/499fc3678b82fbcab2229b8bbec9188e

  puppet apply dpm.pp

Ideally the above command should setup DPM Head node

I have also updated DPM pool nodes rpm packages to latest dpm realease but not the OS. dpm manifest for pool node is here https://gist.github.com/kashif74/3b1a941933bc3688c896fc965859c011

Post Configuration

DNS requests

Our IT services complained that DPM Head node is making too many DNS request. Added all pool nodes in /etc/hosts file of headnode and headnode in /etc/hosts file of poolnodes

Installed nscd for local dns caching

xrootd

latest version of xrootd > 4.8.0. is not in UMD4 so it need to be pulled from EPEL (Feb 2018) . This might resolve in future.

Atlas and CMS name2name library

For CMS : https://twiki.cern.ch/twiki/bin/view/DPM/DPMComponents_Dpm-Xrootd#CMS_federation

For ATLAS : https://twiki.cern.ch/twiki/bin/view/DPM/DPMComponents_Dpm-Xrootd#ATLAS_federation. Atlas rpm is not available from Centos7 repo so install it from SL6 repo.