Difference between revisions of "RAL Tier1 weekly operations castor 11/05/2015"
From GridPP Wiki
(Created page with "[https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_castor List of CASTOR meetings] == Operations News == * ATLAS draining complete * SL6 disk server config is product...") |
|||
Line 60: | Line 60: | ||
* Rob to book meeting to discuss possible workarounds/investigations for CMS issue - xroot timeout / server(read-ahead) or Network tuning / CASTOR bug / deploying more disk servers | * Rob to book meeting to discuss possible workarounds/investigations for CMS issue - xroot timeout / server(read-ahead) or Network tuning / CASTOR bug / deploying more disk servers | ||
* Rob to try and reproduce SRM DB Dups issue with FTS transfers | * Rob to try and reproduce SRM DB Dups issue with FTS transfers | ||
− | |||
− | |||
* Rob - change checksum validator so it maintains case when displaying | * Rob - change checksum validator so it maintains case when displaying | ||
* Bruno to document processes to control services previously controlled by puppet | * Bruno to document processes to control services previously controlled by puppet |
Latest revision as of 15:27, 11 May 2015
Contents
Operations News
- ATLAS draining complete
- SL6 disk server config is production ready, but will not be deployed until DB 11.2.0.4 upgrade is done.
- 2 SL6 tape servers now running in production.
- Testing CASTOR rebalancer on preproduction.
Operations Problems
- CMS CASTOR issues affecting batch farm efficiency - several areas of investigation / mitigation underway
- Seeing a high volume of SRM DB file dup errors, Rob is currently investigating (trying to reproduce with FTS transfers)
- Retrieval errors from facilities castor - a number of similar incidents seen in last few weeks
- log rotate issue - transfer manager logs for rebalancing on CASTOR stagers
- GDSS757 cmsDisk / 763 atlasStripInput - new motherboards, fabric acceptance test complete, ready for deployment.
- Elastic tape data corruption - CASTOR team assisting with extraction of data for review
- A higher number of checksum errors (Alice) investigation ongoing
- OLDER>>>>
- Deadlocks on atlas stager DB (post 2.1.14-15) - not serious but investigation ongoing
- Issue with missing files in LHCb – race condition reported to CERN
- Atlas are putting files (sonar.test files) into un-routable paths - this looks like an issue with the space token used. Brian / Rob actions below.
- There is a problem with latest gfal libraries - not a new problem but Atlas are starting to exercise functionality and identifying these issues
- storageD retrieval from castor problems - investigation ongoing
- castor functional test on lcgccvm02 causing problems - Gareth reviewing.
Blocking Issues
- grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.
Planned, Scheduled and Cancelled Interventions
Advanced Planning
Tasks
- Create new small VOs for DiRAC (and LIGO?)
- Bruno working on SL6 disk servers
- Provide new VM? to provide castor client functionality to query the backup DBs
- Plan to ensure PreProd represents production in terms of hardware generation are underway
- Switch from admin machines: lcgccvm02 to lcgcadm05
- Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers
- Intervention to upgrade CASTOR DBs to Oracle 11.2.0.4
Interventions
Staffing
- Castor on Call person next week
- Shaun
- Staff absence/out of the office:
- Juan out from Tues
- Jens out Monday/Tues
- Shaun not available on Wed
Actions
- Rob and Matt to escalate CASTOR slowness to Massimo at CERN
- Shaun to send plots to Matt to support above action
- Rob to book meeting to discuss possible workarounds/investigations for CMS issue - xroot timeout / server(read-ahead) or Network tuning / CASTOR bug / deploying more disk servers
- Rob to try and reproduce SRM DB Dups issue with FTS transfers
- Rob - change checksum validator so it maintains case when displaying
- Bruno to document processes to control services previously controlled by puppet
- Gareth to arrange meeting castor/fab/production to discuss the decommissioning procedures
- Gareth to investigate providing checks for /etc/noquatto on production nodes & checks for fetch-crl
Completed Actions
- Brian - On Wed, tell Tim if he can start repacking mctape (Done)
- Rob - talk with Shaun to see if its possible to reject anything that does not have a spacetoken (answer: this is not a good idea)
- Brian - to discuss unroutable files / spacetokens at at DDM meeting on Tuesday (Done)
- Rob to pick up DB cleanup change control (Done)
- Chris/Rob to arrange a meeting to discuss CMS performance/xroot issues (is performance appropriate, if not plan to resolve) - inc. Shaun, Rob, Brian, Gareth (Potential fix implemented)