RAL Tier1 weekly operations castor 22/12/2014
From GridPP Wiki
Revision as of 11:30, 19 December 2014 by Christopher Prosser 1e304264ea (Talk | contribs)
Contents
Operations News
- ATLAS atlasStripInput is still very full.
- Draining - latest estimate is to complete draining in 7 weeks, once it resumes.
- Kernel and errata upgrade on Castor SL6 headnodes (including reboot) - Tues 23rd 10:00 - 12:00
Operations Problems
- Switch and DNS problems last Saturday caused a significant outage, however not aware of any fallout from this.
- CMS are experiencing poor performance with xroot stress test - possibly caused by existing heavy load or possibly xroot redirector? Andrew will retry.
- cedaRetrieve issue
- RAL Ops SAM tests (lcgccvm02) have been fixed (initially causing false positives) - initally failed as a side affect of SL6 headnode upgrade.
- GDSS778 (lhcbDst) out for memory failure – now back in.
- LHCb SRM user file table dups – only a few and now removed.
- gdss720 / gdss763 are both drained, out of production and currently being worked on by Fabric team.
- gdss659 is still but will be decommissioned out of CASTOR.
Blocking Issues
- grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.
Planned, Scheduled and Cancelled Interventions
- Kernel and errata upgrade on Castor SL6 headnodes (including reboot) - Tues 23rd 10:00 - 12:00
- A Tier 1 Database cleanup is planned so as to eliminate a number of excess tables and other entities left over from previous CASTOR versions. This will be change-controlled in the near future.
- Upgrade of CASTOR headnodes to SL6 CMS Tuesday 9/12/14 and Atlas Wednesday 10/12/14 and Gen W/C 5th Jan
- Upgrade Oracle DB to version 11.2.0.4 (Late February?)
- Upgrade CASTOR to version 2.1.14-14 OR 2.1.14-15 (Early February)
Advanced Planning
Tasks
- DB team need to plan some work which will result in the DBs being under load for approx 1h - not terribly urgent but needs to be done in new year.
- Provide new VM? to provide castor client functionality to query the backup DBs
- Plan to ensure PreProd represents production in terms of hardware generation are underway
- Possible future upgrade to CASTOR 2.1.14-15 post-Christmas
- Switch from admin machines: lcgccvm02 to lcgcadm05
- Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers
Interventions
Actions
Staffing
- Castor on Call person
- Matt to 24/12
- Rob 25/12-29/12
- Matt from 30/12 – 4/1
- Staff absence/out of the office:
- Only CASTOR person in office Mon-Wed is Matt