RAL Tier1 weekly operations castor 04/05/2015
From GridPP Wiki
Contents
Operations News
- ATLAS draining very nearly complete.
- Upgrade of CASTOR DBs to Oracle version DB 11.2.04 complete.
- SL6 disk server config is production ready, but will not be deployed until DB 11.2.0.4 upgrade is done.
- 2 SL6 tape servers now running in production.
- Testing CASTOR rebalancer on preproduction.
- A potential fix for the CMS xroot access failures has been implemented. This will not fix the slow file open times, but will stop the intermittent job failures.
Operations Problems
- GDSS757 cmsDisk / 763 atlasStripInput - new motherboards, fabric acceptance test complete, ready for deployment.
- Elastic tape data corruption - CASTOR team assisting with extraction of data for review
- Seeing a high volume of SRM DB file dup errors - produced a template for a separated test which will avoid calling on minor versions on this.
- A higher number of checksum errors (Alice) investigation ongoing
- OLDER>>>>
- Deadlocks on atlas stager DB (post 2.1.14-15) - not serious but investigation ongoing
- Issue with missing files in LHCb – race condition reported to CERN
- Atlas are putting files (sonar.test files) into un-routable paths - this looks like an issue with the space token used. Brian / Rob actions below.
- There is a problem with latest gfal libraries - not a new problem but Atlas are starting to exercise functionality and identifying these issues
- storageD retrieval from castor problems - investigation ongoing
- castor functional test on lcgccvm02 causing problems - Gareth reviewing.
Blocking Issues
- grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.
Planned, Scheduled and Cancelled Interventions
Advanced Planning
Tasks
- Bruno working on SL6 disk servers
- Provide new VM? to provide castor client functionality to query the backup DBs
- Plan to ensure PreProd represents production in terms of hardware generation are underway
- Switch from admin machines: lcgccvm02 to lcgcadm05
- Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers
- Intervention to upgrade CASTOR DBs to Oracle 11.2.0.4
Interventions
Staffing
- Castor on Call person next week
- Chris
- Staff absence/out of the office:
- n/a
Actions
- Rob - revisit deployment plans (esp people in office towards the end of March)
- Rob - change checksum validator so it maintains case when displaying
- Bruno to document processes to control services previously controlled by puppet
- Gareth to arrange meeting castor/fab/production to discuss the decommissioning procedures
- Gareth to investigate providing checks for /etc/noquatto on production nodes & checks for fetch-crl
Completed Actions
- Brian - On Wed, tell Tim if he can start repacking mctape (Done)
- Rob - talk with Shaun to see if its possible to reject anything that does not have a spacetoken (answer: this is not a good idea)
- Brian - to discuss unroutable files / spacetokens at at DDM meeting on Tuesday (Done)
- Rob to pick up DB cleanup change control (Done)
- Chris/Rob to arrange a meeting to discuss CMS performance/xroot issues (is performance appropriate, if not plan to resolve) - inc. Shaun, Rob, Brian, Gareth (Potential fix implemented)