Difference between revisions of "RAL Tier1 weekly operations castor 08/09/2014"

From GridPP Wiki
Jump to: navigation, search
(Created page with "== Operations News == * Plan to ensure PreProd represents production in terms of hardware generation are underway * Disk server redeployments continue (i.e. D1T0 reused in D0T...")
 
 
Line 6: Line 6:
  
 
== Operations Problems ==
 
== Operations Problems ==
* a couple of disk server issues (GDSS748 and GDSS659).
+
* a couple of disk server issues (GDSS748 and GDSS659)
* Gen transfermanager on lsf03 failed 00:03 Fri 5th, restarted by Puppet.
+
* Gen transfermanager on lsf03 failed 00:03 Fri 5th, restarted by Puppet
* var partition on headnodes filling up with rsyslogs as DLF is no longer there to pick them up - Rob has corrected this.
+
* var partition on headnodes filling up with rsyslogs as DLF is no longer there to pick them up - Rob has corrected this
 
* Brian has found atlas scratch files on tape caches - concern about d0t1 decommissioning procedure, Brian to propose/update
 
* Brian has found atlas scratch files on tape caches - concern about d0t1 decommissioning procedure, Brian to propose/update
  
 
== Blocking Issues ==
 
== Blocking Issues ==
 
* LHCb tape cache deployment – gdss673 not installing
 
* LHCb tape cache deployment – gdss673 not installing
* Juan – patch castor dbs beginning of Nov PSU patches – standard change
 
 
* grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.
 
* grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.
  

Latest revision as of 12:37, 5 September 2014

Operations News

  • Plan to ensure PreProd represents production in terms of hardware generation are underway
  • Disk server redeployments continue (i.e. D1T0 reused in D0T1 etc) ... final few now
  • Tim has been busy migrating CMS data to D tapes
  • SL6 Headnode work progressing - hoping for rollout in Nov

Operations Problems

  • a couple of disk server issues (GDSS748 and GDSS659)
  • Gen transfermanager on lsf03 failed 00:03 Fri 5th, restarted by Puppet
  • var partition on headnodes filling up with rsyslogs as DLF is no longer there to pick them up - Rob has corrected this
  • Brian has found atlas scratch files on tape caches - concern about d0t1 decommissioning procedure, Brian to propose/update

Blocking Issues

  • LHCb tape cache deployment – gdss673 not installing
  • grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.

Planned, Scheduled and Cancelled Interventions

  • Juan to patch castor dbs beginning of Nov PSU patches – standard change
  • 2.1.14-14 testing in preprod when Brian releases it
  • A Tier 1 Database cleanup is planned so as to eliminate a number of excess tables and other entities left over from previous CASTOR versions. This will be change-controlled in the near future.


Advanced Planning

Tasks

  • Possible future upgrade to CASTOR 2.1.14-15.
  • Switch from admin machines: lcgccvm02 to lcgcadm05
  • New VM configured to run against the standby CASTOR database will be created as a front-end for dark data etc queries.
  • Replace DLF with Elastic Search
  • Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers


Interventions

  • None

Staffing

  • Castor on Call person
    • Rob


  • Staff absence/out of the office:
    • All in