RAL Tier1 weekly operations castor 12/01/2015

Operations News

cmsDisk GDSS757 back in production (failed not responding - no issues found)
Draining - ongoing
GEN SL6 headnode upgrade – success
SL6 name server upgrade postponed due to castor team resource - needs to be rescheduled

LHCb ticket investigation - FTS problem, double prepare to put resulting in zero length files - has hopefully been fixed by a past upgrade. We need to follow up by investigating time periods and provide lists of files to VOs (190k total across all LHC VOs)
Heavy LHCb usage causing significant castor load - SRM test failures. Number of jobs reduced for the weekend.
Ganglia monitoring - atlas missing some data e.g. ganglia. Castor to raise a ticket in support.
Checksum Mismatch - Facilities tape issues, Tim hoping to retrieve problem file from offsite tape

grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.

Kernel upgrade on Castor SL5 disk/srm/tape. Currently in planning but likely to be last week in Jan
A Tier 1 Database cleanup is planned so as to eliminate a number of excess tables and other entities left over from previous CASTOR versions. This will be change-controlled in the near future.
Upgrade of CASTOR headnodes to SL6 Gen W/C 5th Jan
Upgrade Oracle DB to version 11.2.0.4 (Late February?)
Upgrade CASTOR to version 2.1.14-14 OR 2.1.14-15 (Early February)

Tasks

DB team need to plan some work which will result in the DBs being under load for approx 1h - not terribly urgent but needs to be done in new year.
Provide new VM? to provide castor client functionality to query the backup DBs
Plan to ensure PreProd represents production in terms of hardware generation are underway
Possible future upgrade to CASTOR 2.1.14-15 post-Christmas
Switch from admin machines: lcgccvm02 to lcgcadm05
Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers

Interventions