RAL Tier1 weekly operations castor 15/12/2014

From GridPP Wiki
Jump to: navigation, search

List of CASTOR meetings

Operations News

  • ATLAS have filled their available space in atlasStripInput, so we have paused draining pending deletions (currently 3% or 175.21T free).
  • Draining - latest estimate is to complete draining in 7 weeks, assuming no breaks.
  • SL6 has been rolled out to CMS and Atlas (LHCB previously) headnodes, no serious issues encountered


Operations Problems

  • gdss720 / gdss763 are both drained, out of production and currently being worked on by Fabric team
  • gdss659 is still but will be decommissioned out of CASTOR.
  • CMS xroot redirector has been moved from lcgclsf02 to another temporary node (was causing issues on LSF02).
  • Atlas srmbed's segfaulting due to // in file paths, workaround that had previously been applied to Gen was also applied to Atlas, CMS and LHCb this week.
  • RAL Ops tests failing post SL6 upgrade, root cause found to be user account that was used being removed from headnodes - fix on its way.


Blocking Issues

  • grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.

Planned, Scheduled and Cancelled Interventions

  • A Tier 1 Database cleanup is planned so as to eliminate a number of excess tables and other entities left over from previous CASTOR versions. This will be change-controlled in the near future.
  • Upgrade of CASTOR headnodes to SL6 CMS Tuesday 9/12/14 and Atlas Wednesday 10/12/14 and Gen W/C 5th Jan
  • Upgrade Oracle DB to version 11.2.0.4 (Late February?)
  • Upgrade CASTOR to version 2.1.14-14 OR 2.1.14-15 (Early February)


Advanced Planning

Tasks

  • DB team need to plan some work which will result in teh DBs being under load for approx 1h - not terribly urgent but needs to be done in new year.
  • Plan and publish SL6 deployment plans
  • Plan to ensure PreProd represents production in terms of hardware generation are underway
  • Possible future upgrade to CASTOR 2.1.14-15 post-Christmas
  • Switch from admin machines: lcgccvm02 to lcgcadm05
  • Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers

Interventions


Actions

Staffing

  • Castor on Call person
    • Matt to 24/12
    • Rob 25/12 -29/12
    • Matt from 30/12 – 4/1


  • Staff absence/out of the office:
    • Rob / Shaun - Out until 2015 (however Rob doing oncall as above)
    • Chris out Tuesday / in late Wed
    • Juan out Wed - 8th Jan
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=RAL_Tier1_weekly_operations_castor_15/12/2014&oldid=7374"