RAL Tier1 weekly operations castor 01/09/2014
From GridPP Wiki
Revision as of 15:06, 29 August 2014 by Christopher Prosser 1e304264ea (Talk | contribs)
Contents
Operations News
- Samneet's web query tool is under development, the alpha version is now available for use (http://www-private.gridpp.rl.ac.uk/elasticquery/CurrentESQP.html).
- Plan to ensure PreProd represents production in terms of hardware generation are underway.
- Disk server redeployments continue (i.e. D1T0 reused in D0T1 etc)
- We have been draining a small number of disk servers without incident following rollout of a database fix last week (to all Tier 1 not facilities)
- Oracle patch applied to preprod last Wednesday
Operations Problems
- Load issue on AtlasScratchDisk last weekend causing failing SAM / Functional tests and poor performance.
- Duplicate files in the CMS SRM database - contacted CMS representatives and clearing up as they occur.
- a couple of disk server issues BAU
- passive draining produces file duplication - fixed in castor 2.1.14-14
- The rebalancer has been tested and found to cause problematically large transfermanager queues even with low thresholds set. We will not be using it further until we have a fix.
Blocking Issues
- SL6 castor stalled due to resource limitations & A/L
- grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.
Planned, Scheduled and Cancelled Interventions
- 2.1.14-14 testing in preprod
- A Tier 1 Database cleanup is planned so as to eliminate a number of excess tables and other entities left over from previous CASTOR versions. This will be change-controlled in the near future.
Advanced Planning
Tasks
- Possible future upgrade to CASTOR 2.1.14-15.
- Come up with an SL6 configuration of CASTOR nodes implemented using Aquilon.
- Resume draining on the ATLAS instance once draining issues resolved.
- Switch from admin machines: lcgccvm02 to lcgcadm05
- New VM configured to run against the standby CASTOR database will be created as a front-end for dark data etc queries.
- Replace DLF with Elastic Search
- Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers
Interventions
- None
Staffing
- Castor on Call person
- Rob
- Staff absence/out of the office:
- Bruno out Monday/Tues