RAL Tier1 weekly operations castor 22/12/2014

From GridPP Wiki
Jump to: navigation, search

List of CASTOR meetings

Operations News

  • ATLAS atlasStripInput is still very full.
  • Draining - latest estimate is to complete draining in 7 weeks, once it resumes.
  • Kernel and errata upgrade on Castor SL6 headnodes (including reboot) - Tues 23rd 10:00 - 12:00

Operations Problems

  • Switch and DNS problems last Saturday caused a significant outage, however not aware of any fallout from this.
  • CMS are experiencing poor performance with xroot stress test - possibly caused by existing heavy load or possibly xroot redirector? Andrew will retry.
  • cedaRetrieve issue
  • RAL Ops SAM tests (lcgccvm02) have been fixed (initially causing false positives) - initally failed as a side affect of SL6 headnode upgrade.
  • GDSS778 (lhcbDst) out for memory failure – now back in.
  • LHCb SRM user file table dups – only a few and now removed.
  • gdss720 / gdss763 are both drained, out of production and currently being worked on by Fabric team.
  • gdss659 is still but will be decommissioned out of CASTOR.


Blocking Issues

  • grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.

Planned, Scheduled and Cancelled Interventions

  • Kernel and errata upgrade on Castor SL6 headnodes (including reboot) - Tues 23rd 10:00 - 12:00
  • A Tier 1 Database cleanup is planned so as to eliminate a number of excess tables and other entities left over from previous CASTOR versions. This will be change-controlled in the near future.
  • Upgrade of CASTOR headnodes to SL6 Gen W/C 5th Jan
  • Upgrade Oracle DB to version 11.2.0.4 (Late February?)
  • Upgrade CASTOR to version 2.1.14-14 OR 2.1.14-15 (Early February)


Advanced Planning

Tasks

  • DB team need to plan some work which will result in the DBs being under load for approx 1h - not terribly urgent but needs to be done in new year.
  • Provide new VM? to provide castor client functionality to query the backup DBs
  • Plan to ensure PreProd represents production in terms of hardware generation are underway
  • Possible future upgrade to CASTOR 2.1.14-15 post-Christmas
  • Switch from admin machines: lcgccvm02 to lcgcadm05
  • Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers

Interventions


Actions

Staffing

  • Castor on Call person
    • Matt to 24/12
    • Rob 25/12-29/12
    • Matt from 30/12 – 4/1


  • Staff absence/out of the office:
    • Only CASTOR person in office Mon-Wed is Matt
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=RAL_Tier1_weekly_operations_castor_22/12/2014&oldid=7050"