RAL Tier1 weekly operations castor 29/02/2016

Operations News

No disk server issues this week
glibc updates applied, all CASTOR systems rebooted. initial issues with head nodes, 7 failed to reboot due to their build history. ACTION: RA to revisit quattor build so that this does not recur.
11.2.0.4 DB client update had to be rescheduled, should go ahead Monday 29th, has been running in pre-prod for considerable amount of time. This should be transparent. ACTION RA

Main CIP system failed, have failed over to test CIP machine. HW failure to be fixed then will fail back over to production system. ACTION: RA, CC and Fabric
OPN links. BD investigating what data flow is filling OPN and superjanet at the same time.
LHCb job failures - GGUS ticket open
ongoing AAA issues in CMS
CV '11 gen RAID card controller firmware update required - WD digital drives failing at a high rate ACTION: RA follow up with fabric team

2.1.15 update to nameserver will not go ahead. This is due to slow file open times issues on the stager. Testing/debugging of stager issue is ongoing. If these issues are resolved propose nameserver upgrade on 16th March with stager following week, 22nd March. (RA)
11.2.04 client updates (running in preprod) - possible change control for prod (see above)
WAN tuning proposal - possibly put into change control BD
CASTOR facilities patching scheduled for next week - detailed schedule to be agreed with fabric team.

RA has produced a python script to handle SRM db duplication issue which is causing callouts. This script has been tested and will now be put into production, as a cron job. This should be a temporary fix, so a bug report should be made to the FTS development team, via AL.
JJ – Glue 2 for CASTOR, used for publishing information. RA writing data getting end in python, JJ writing Glue 2 end in LISP. No schedule as yet.
Facilities drive re-allocation. ACTION: RA
SRM 2.1.14 testing with SdW on VCERT

Tasks

Interventions

Staff absence/out of the office:
- BD - Monday-Friday
- CP - Monday-Tuesday
- SdW - Tuesday-Wednesday

RA to revisit quattor build for the head nodes which did not reboot so that this does not recur.
RA 11.2.0.4 DB client update had to be rescheduled, should go ahead Monday 29th
RA, CC and Fabric - fix CIP production system and switch back from test server
RA follow up with Fabric re: CV '11 gen RAID card controller firmware update

BD to coordinate with atlas re bulk deletion before TF starts repack
GS arrange a meeting to discuss remaining actions on CV11 and V12 (when KH is back)
BD to clarify if separating the DiRAC data is a necessity
BD ensure quattorising atlas consistency check
SdW to send merging tape pools wiki to CERN for review
RA to deploy a 14 generation into preprod
BD re. WAN tuning proposal - discuss with GS, does it need a change control?
RA to try stopping tapeserverd mid-migration to see if it breaks.
RA (was SdW) to modify cleanlostfiles to log to syslog so we can track its use - under testing
GS to investigate how/if we need to declare xrootd endpoints in GOCDB BDII - progress