RAL Tier1 Operations Report for 10th May 2017
| Review of Issues during the week 3rd to 10th May 2017.
|
- LHCb have managed to complete their stripping and merging campaign for data stored at RAL. Nevertheless we still need to follow up with improvements to Castor to resolve throughput and stability issues. The SRMs required a restart on Friday aftternoon (28th April).
- AtlasDataDisk became full. This severely affected the Atlas SAM tests and hence availability. Investigations show problems with Castor's file deletion rate. Once the disk is full any attempted writes are subsequently cleaned up (- i.e. a delete request). These deletions take all of Castor's capacity to delete files. The result is that once a disk area fills the problem becomes a severe fault and is only resolved by the VO stopping attempts to write data - allowing some level of deletions to happen. However, it is clear that the deletion rate was not sufficient at times before the disk became full - although this is not understood.
- There was a problem with Castor for CMS overnight last night (Tuesday/Wednesday 25/26 April) when the database reported blocking sessions. This fixed itself after a couple of hours.
- There is an ongoing problem with one of the argus servers. This has affected CMS CE SAM tests (and hence availability) since yesterday (Tuesday) evening. Work is ongoing at the time of the meeting.
- We are also seeing a non-negligible failure rate for CE SAM tests for CMS that is not yet understood. CMS are experimented in running with 'lazy download' disabled.
- There is a problem with the UPS system in the R89 computer problem. Internal capacitors overheated last Friday (28th April) and we are currently running in Mains by-Pass without UPS protection.
| Resolved Disk Server Issues
|
- GDSS793 (AtlasDataDisk - D1T0) was taken out of service on Friday 5th May with a double disk failure. It was retured to service on Monday (8th) after thw first disk replacement had comleted rebuilding.
- GDSS818 (LHCbDst - D1T0) failed twice in this last week. The first time was in the early hours of 5th May. During that day its RAID card was swapped and it was returned to service that afternoon (read-only). It failed again during the following night. Following an intervention over the weekend the server was put back read-only on Monday (8th May) and will be drained out pending further investigation.
| Current operational status and issues
|
- We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities.
| Ongoing Disk Server Issues
|
| Limits on concurrent batch system jobs.
|
| Notable Changes made since the last meeting.
|
- Updated GridFTP / XRootD plugins on ECHO Ceph gateways changing the way checksum is stored.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| srm-lhcb.gridpp.rl.ac.uk
|
SCHEDULED
|
OUTAGE
|
11/05/2017 10:00
|
11/05/2017 16:00
|
6 hours
|
Downtime while upgrading LHCb Castor instance to 2.1.16
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
Pending - but not yet formally announced:
- Update Castor (including SRMs) to version 2.1.16. Central nameserver done. Current plan: LHCb stager on Thursday 11th May. Others to follow.
- Update Castor SRMs - CMS & GEN still to do. Problems seen with the SRM update mean these will wait until Castor 2.1.16 is rolled out.
Listing by category:
- Castor:
- Update SRMs to new version, including updating to SL6.
- Update to version 2.1.16
- Merge AtlasScratchDisk into larger Atlas disk pool.
- Networking
- Enable first services on production network with IPv6 now that the addressing scheme has been agreed. (Perfsonar already working over IPv6).
| Entries in GOC DB starting since the last report.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| srm-lhcb.gridpp.rl.ac.uk
|
SCHEDULED
|
OUTAGE
|
09/05/2017 10:00
|
09/05/2017 12:22
|
2 hours and 22 minutes
|
Update central Castor services (including the nameserver component) and LHCb stager to version 2.1.16
|
| All Castor (except LHCb)
|
SCHEDULED
|
OUTAGE
|
09/05/2017 10:00
|
09/05/2017 12:21
|
2 hours and 21 minutes
|
Update central Castor services (including the nameserver component) to version 2.1.16
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 128180
|
Green
|
Urgent
|
In Progress
|
2017-05-05
|
2017-05-08
|
|
WLGC-IPv6 Tier-1 readiness
|
| 128068
|
Green
|
Urgent
|
On Hold
|
2017-05-04
|
2017-05-04
|
CMS
|
T1_UK_RAL, SAM tests failing
|
| 127968
|
Green
|
Less Urgent
|
In Progress
|
2017-04-27
|
2017-04-27
|
MICE
|
RAL castor: not able to list directories and copy to
|
| 127967
|
Green
|
Less Urgent
|
On Hold
|
2017-04-27
|
2017-04-28
|
MICE
|
Enabling pilot role for mice VO at RAL-LCG2
|
| 127916
|
Yellow
|
Alarm
|
In Progress
|
2017-04-25
|
2017-04-25
|
LHCb
|
RAL srm-s are down again for LHCb
|
| 127612
|
Red
|
Alarm
|
In Progress
|
2017-04-08
|
2017-05-09
|
LHCb
|
CEs at RAL not responding
|
| 127598
|
Green
|
Urgent
|
In Progress
|
2017-04-07
|
2017-04-07
|
CMS
|
UK XRootD Redirector
|
| 127597
|
Yellow
|
Urgent
|
Waiting for Reply
|
2017-04-07
|
2017-05-04
|
CMS
|
Check networking and xrootd RAL-CERN performance
|
| 127240
|
Amber
|
Urgent
|
In Progress
|
2017-03-21
|
2017-05-08
|
CMS
|
Staging Test at UK_RAL for Run2
|
| 124876
|
Red
|
Less Urgent
|
On Hold
|
2016-11-07
|
2017-01-01
|
OPS
|
[Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
|
| 117683
|
Red
|
Less Urgent
|
On Hold
|
2015-11-18
|
2017-03-02
|
|
CASTOR at RAL not publishing GLUE 2.
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC ECHO = Atlas ECHO (Template 841);CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas ECHO |
Atlas HC |
Atlas HC ECHO |
CMS HC |
Comment
|
| 03/05/17 |
100 |
100 |
87 |
36 |
96 |
100 |
98 |
100 |
85 |
CMS: glexec test failures; Others SRM test failures.
|
| 04/05/17 |
100 |
100 |
94 |
100 |
96 |
100 |
98 |
99 |
93 |
SRM test failures.
|
| 05/05/17 |
100 |
100 |
96 |
100 |
100 |
100 |
100 |
100 |
100 |
SRM test failures on GET ("unknown error")
|
| 06/05/17 |
100 |
100 |
81 |
100 |
100 |
100 |
100 |
100 |
N/A |
SRM test failures on GET ("unknown error")
|
| 07/05/17 |
100 |
100 |
85 |
96 |
100 |
100 |
100 |
100 |
100 |
SRM test failures: Atlas - on GET ("unknown error"); CMS - Unable to issue PrepareToPut/Get request
|
| 08/05/17 |
100 |
100 |
98 |
100 |
100 |
100 |
100 |
100 |
N/A |
Single SRM test failure on GET ("unknown error")
|
| 09/05/17 |
100 |
100 |
77 |
55 |
87 |
100 |
90 |
98 |
100 |
All: Castor planned outage for a couple of hours. Atlas: Plus sporadic SRM test failures (unknown error); CMS CE tests failed owing to xroot redirection failing.
|
