Difference between revisions of "RAL Tier1 weekly operations castor 16/02/2015"
From GridPP Wiki
(Created page with " IN PROGRESS ..... [https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_castor List of CASTOR meetings] == Operations News == * Draining - Atlas now draining again /...") |
|||
| Line 1: | Line 1: | ||
| − | |||
| − | |||
| − | |||
| − | |||
[https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_castor List of CASTOR meetings] | [https://www.gridpp.ac.uk/wiki/RAL_Tier1_weekly_operations_castor List of CASTOR meetings] | ||
== Operations News == | == Operations News == | ||
| − | * Draining - | + | * Draining - LHCb currently draining |
| − | + | ||
| − | + | ||
== Operations Problems == | == Operations Problems == | ||
* There is a problem with latest gfal libraries - not a new problem but Atlas are starting to exercise functionality and identifying these issues | * There is a problem with latest gfal libraries - not a new problem but Atlas are starting to exercise functionality and identifying these issues | ||
| − | * Atlas are putting files (sonar.test files) into un-routable paths - this looks like an issue with the space token used | + | * Atlas are putting files (sonar.test files) into un-routable paths - this looks like an issue with the space token used. Brian / Rob. |
| + | |||
* storageD retrieval from castor problems - investigation ongoing | * storageD retrieval from castor problems - investigation ongoing | ||
* CMS heavy load & high job failure – hot spotting 3 servers, files spread to many servers which then also became loaded. CMS moved away from RAL for these files. Need to discuss this issue in more detail. | * CMS heavy load & high job failure – hot spotting 3 servers, files spread to many servers which then also became loaded. CMS moved away from RAL for these files. Need to discuss this issue in more detail. | ||
| Line 26: | Line 21: | ||
== Planned, Scheduled and Cancelled Interventions == | == Planned, Scheduled and Cancelled Interventions == | ||
| − | |||
| − | |||
* Upgrade Oracle DB to version 11.2.0.4 (Late February?) | * Upgrade Oracle DB to version 11.2.0.4 (Late February?) | ||
* Upgrade CASTOR to version 2.1.14-14 OR 2.1.14-15 (February) | * Upgrade CASTOR to version 2.1.14-14 OR 2.1.14-15 (February) | ||
| Line 34: | Line 27: | ||
== Advanced Planning == | == Advanced Planning == | ||
'''Tasks''' | '''Tasks''' | ||
| + | * SL6 tape server | ||
| + | * Bruno working on SL6 disk servers | ||
* DB team need to plan some work which will result in the DBs being under load for approx 1h - not terribly urgent but needs to be done in new year. | * DB team need to plan some work which will result in the DBs being under load for approx 1h - not terribly urgent but needs to be done in new year. | ||
* Provide new VM? to provide castor client functionality to query the backup DBs | * Provide new VM? to provide castor client functionality to query the backup DBs | ||
| Line 45: | Line 40: | ||
== Actions == | == Actions == | ||
| + | * Rob - talk with shaun to see if its possible to reject anything that does not have a spacetoken | ||
| + | * Brian - to discuss unroutable files / spacetokens at at DDM meeting on Tuesday | ||
* Rob to pick up DB cleanup change control | * Rob to pick up DB cleanup change control | ||
* Bruno to document processes to control services previously controlled by puppet | * Bruno to document processes to control services previously controlled by puppet | ||
| Line 50: | Line 47: | ||
* Chris/Rob to arrange a meeting to discuss CMS performance/xroot issues (is performance appropriate, if not plan to resolve) - inc. Shaun, Rob, Brian, Gareth | * Chris/Rob to arrange a meeting to discuss CMS performance/xroot issues (is performance appropriate, if not plan to resolve) - inc. Shaun, Rob, Brian, Gareth | ||
* Gareth to investigate providing checks for /etc/noquatto on production nodes & checks for fetch-crl | * Gareth to investigate providing checks for /etc/noquatto on production nodes & checks for fetch-crl | ||
| − | |||
| − | |||
== Staffing == | == Staffing == | ||
* Castor on Call person | * Castor on Call person | ||
| − | ** Monday - Friday | + | ** Monday-Friday Chris |
| − | ** Weekend | + | ** Weekend TBA |
* Staff absence/out of the office: | * Staff absence/out of the office: | ||
** | ** | ||
Revision as of 15:25, 16 February 2015
Contents
Operations News
- Draining - LHCb currently draining
Operations Problems
- There is a problem with latest gfal libraries - not a new problem but Atlas are starting to exercise functionality and identifying these issues
- Atlas are putting files (sonar.test files) into un-routable paths - this looks like an issue with the space token used. Brian / Rob.
- storageD retrieval from castor problems - investigation ongoing
- CMS heavy load & high job failure – hot spotting 3 servers, files spread to many servers which then also became loaded. CMS moved away from RAL for these files. Need to discuss this issue in more detail.
- castor functional test on lcgccvm02 causing problems - Gareth reviewing
- 150k zero size files reported last week have almost all been dealt with, CMS files outstanding
- Files with no ns or xattr checksum value in castor are failing transfers from RAL to BNL using the BNL FTS3 server.
Blocking Issues
- grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.
Planned, Scheduled and Cancelled Interventions
- Upgrade Oracle DB to version 11.2.0.4 (Late February?)
- Upgrade CASTOR to version 2.1.14-14 OR 2.1.14-15 (February)
Advanced Planning
Tasks
- SL6 tape server
- Bruno working on SL6 disk servers
- DB team need to plan some work which will result in the DBs being under load for approx 1h - not terribly urgent but needs to be done in new year.
- Provide new VM? to provide castor client functionality to query the backup DBs
- Plan to ensure PreProd represents production in terms of hardware generation are underway
- Possible future upgrade to CASTOR 2.1.14-15 post-Christmas
- Switch from admin machines: lcgccvm02 to lcgcadm05
- Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers
Interventions
Actions
- Rob - talk with shaun to see if its possible to reject anything that does not have a spacetoken
- Brian - to discuss unroutable files / spacetokens at at DDM meeting on Tuesday
- Rob to pick up DB cleanup change control
- Bruno to document processes to control services previously controlled by puppet
- Gareth to arrange meeting castor/fab/production to discuss the decommissioning procedures
- Chris/Rob to arrange a meeting to discuss CMS performance/xroot issues (is performance appropriate, if not plan to resolve) - inc. Shaun, Rob, Brian, Gareth
- Gareth to investigate providing checks for /etc/noquatto on production nodes & checks for fetch-crl
Staffing
- Castor on Call person
- Monday-Friday Chris
- Weekend TBA
- Staff absence/out of the office:
