RAL Tier1 weekly operations castor 29/02/2016
From GridPP Wiki
Contents
Operations News
- No disk server issues this week
- glibc updates applied, all CASTOR systems rebooted. initial issues with head nodes, 7 failed to reboot due to their build history. ACTION: RA to revisit quattor build so that this does not recur.
- 11.2.0.4 DB client update had to be rescheduled, should go ahead Monday 29th, has been running in pre-prod for considerable amount of time. This should be transparent. ACTION RA
Operations Problems
- Main CIP system failed, have failed over to test CIP machine. HW failure to be fixed then will fail back over to production system. ACTION: RA, CC and Fabric
- OPN links. BD investigating what data flow is filling OPN and superjanet at the same time.
- LHCb job failures - GGUS ticket open
- ongoing AAA issues in CMS
- CV '11 gen RAID card controller firmware update required - WD digital drives failing at a high rate ACTION: RA follow up with fabric team
Planned, Scheduled and Cancelled Interventions
- 2.1.15 update to nameserver will not go ahead. This is due to slow file open times issues on the stager. Testing/debugging of stager issue is ongoing. If these issues are resolved propose nameserver upgrade on 16th March with stager following week, 22nd March. (RA)
- 11.2.04 client updates (running in preprod) - possible change control for prod (see above)
- WAN tuning proposal - possibly put into change control BD
- CASTOR facilities patching scheduled for next week - detailed schedule to be agreed with fabric team.
Long-term projects
- RA has produced a python script to handle SRM db duplication issue which is causing callouts. This script has been tested and will now be put into production, as a cron job. This should be a temporary fix, so a bug report should be made to the FTS development team, via AL.
- JJ – Glue 2 for CASTOR, used for publishing information. RA writing data getting end in python, JJ writing Glue 2 end in LISP. No schedule as yet.
- Facilities drive re-allocation. ACTION: RA
- SRM 2.1.14 testing with SdW on VCERT
Advanced Planning
Tasks
- CASTOR 2.1.15 implementation and testing
Interventions
Staffing
- Castor on Call person next week
- RA until Thursday
- Propose CP Friday - TBC
- Staff absence/out of the office:
- BD - Monday-Friday
- CP - Monday-Tuesday
- SdW - Tuesday-Wednesday
New Actions
- RA to revisit quattor build for the head nodes which did not reboot so that this does not recur.
- RA 11.2.0.4 DB client update had to be rescheduled, should go ahead Monday 29th
- RA, CC and Fabric - fix CIP production system and switch back from test server
- RA follow up with Fabric re: CV '11 gen RAID card controller firmware update
Existing Actions
- BD to coordinate with atlas re bulk deletion before TF starts repack
- GS arrange a meeting to discuss remaining actions on CV11 and V12 (when KH is back)
- BD to clarify if separating the DiRAC data is a necessity
- BD ensure quattorising atlas consistency check
- SdW to send merging tape pools wiki to CERN for review
- RA to deploy a 14 generation into preprod
- BD re. WAN tuning proposal - discuss with GS, does it need a change control?
- RA to try stopping tapeserverd mid-migration to see if it breaks.
- RA (was SdW) to modify cleanlostfiles to log to syslog so we can track its use - under testing
- GS to investigate how/if we need to declare xrootd endpoints in GOCDB BDII - progress