RAL Tier1 Operations Report for 15th April 2019
Review of Issues during the week 8th April 2019 to the 15th April 2019.
|
- We are seeing high outbound packet loss over IPv6. Investigations on hold as central networking do not have the expertise (Philip Garrad) available until after Easter.
- High CMS job failure rates. Ongoing issues with meta-data spread across large files. Temporarily limited CMS job slots.
- On Friday 5th April, gdss700 (LHCb) had a double drive failure and needed to be removed from production. Further problems were found, and the while we were able to return the disk to production briefly we were unable to copy all the files off and 1482 were lost.
- On Wednesday 10th April, gdss811 (LHCb) had a failure of the disk running the operating system. This generation of hardware has OS disks that are very inconveniently located (glued to the underside of the motherboard!). Not yet returned to production as of the morning of the 15th.
- On Thursday 11th April, unknown issue caused a significant fraction of docker containers (running jobs) to restart.
Current operational status and issues
|
Resolved Castor Disk Server Issues
|
Machine
|
VO
|
DiskPool
|
dxtx
|
Comments
|
gdss799
|
LHCb
|
lhcb
|
d1t0
|
Machine crashed. It is with fabric at the moment.
|
Ongoing Castor Disk Server Issues
|
Machine
|
VO
|
DiskPool
|
dxtx
|
Comments
|
gdss811
|
LHCb
|
lhcb
|
d1t0
|
Machine crashed. It is with fabric at the moment.
|
Limits on concurrent batch system jobs.
|
Notable Changes made since the last meeting.
|
Entries in GOC DB starting since the last report.
|
Service
|
ID
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
Service
|
ID
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
Listing by category:
- DNS servers will be rolled out within the Tier1 network.
Open
GGUS Tickets (Snapshot taken during morning of the meeting).
|
Request id
|
Affected vo
|
Status
|
Priority
|
Date of creation
|
Last update
|
Type of problem
|
Subject
|
Scope
|
Solution
|
140725
|
cms
|
in progress
|
urgent
|
15/04/2019
|
15/04/2019
|
CMS_Facilities
|
T1_UK_RAL intermittent xrootd relative failures
|
WLCG
|
|
140660
|
cms
|
in progress
|
urgent
|
09/04/2019
|
10/04/2019
|
CMS_Central Workflows
|
FIle read issues for Workflows where data is located at T1_UK_RAL
|
WLCG
|
|
140447
|
dteam
|
in progress
|
less urgent
|
27/03/2019
|
02/04/2019
|
Network problem
|
packet loss outbound from RAL-LCG2 over IPv6
|
EGI
|
|
140220
|
mice
|
in progress
|
less urgent
|
15/03/2019
|
08/04/2019
|
Other
|
mice LFC to DFC transition
|
EGI
|
|
139672
|
other
|
in progress
|
urgent
|
13/02/2019
|
08/04/2019
|
Middleware
|
No LIGO pilots running at RAL
|
EGI
|
|
138033
|
atlas
|
waiting for reply
|
urgent
|
01/11/2018
|
12/04/2019
|
Other
|
singularity jobs failing at RAL
|
EGI
|
|
GGUS Tickets Closed Last week
|
Request id
|
Affected vo
|
Status
|
Priority
|
Date of creation
|
Last update
|
Type of problem
|
Subject
|
Scope
|
Solution
|
140683
|
lhcb
|
solved
|
top priority
|
10/04/2019
|
12/04/2019
|
Local Batch System
|
Pilots failing at RAL across all CEs
|
WLCG
|
Problem was resolved and checks put in place to prevent recurrence.
|
140577
|
lhcb
|
solved
|
less urgent
|
04/04/2019
|
11/04/2019
|
File Access
|
LHCb disk only files requested with the wrong service class
|
EGI
|
No solution found so far. LHCb is close to migrate from the old CASTIR instance soon
|
140385
|
cms
|
closed
|
less urgent
|
25/03/2019
|
12/04/2019
|
CMS_Data Transfers
|
Data Transfer problems at T1_UK_RAL
|
WLCG
|
The datasets listed had each one file with a problem. Two had zero size and were deleted from tape. Another seven were deleted from tape and invalidated by the data transfer team. The list of stuck routing files is now empty apart from a file from a T2 site currently experiencing problems.
|
139723
|
atlas
|
closed
|
less urgent
|
15/02/2019
|
10/04/2019
|
Data Management - generic
|
permissions on scratchdisk
|
EGI
|
Hi Folks,
This appears to have become a somewhat convoluted ticket which as far as I can tell has simply become a conversation thread.
Assuming that the original issue has been resolved (further workarounds/enhancements not withstanding), I'm marking this as solved.
Cheers
Confused of Tier-1! (Darren)
|
138665
|
mice
|
solved
|
urgent
|
04/12/2018
|
09/04/2019
|
Middleware
|
Problem accessing LFC at RAL
|
EGI
|
As I understand it this ticket is has been superseded by https://ggus.eu/?mode=ticket_info&ticket_id=140220. As such I'm closing this ticket. Please feel free to reopen this ticket if you disagree.
|
|
Day
|
Atlas
|
Atlas-Echo
|
CMS
|
LHCB
|
Alice
|
OPS
|
Comments
|
2019-04-08
|
100
|
100
|
100
|
100
|
100
|
100
|
|
2019-04-09
|
100
|
100
|
100
|
100
|
100
|
100
|
|
2019-04-10
|
69
|
69
|
100
|
100
|
100
|
100
|
|
2019-04-11
|
61
|
61
|
100
|
100
|
100
|
100
|
|
2019-04-12
|
100
|
100
|
100
|
100
|
100
|
100
|
|
2019-04-13
|
100
|
100
|
100
|
100
|
100
|
100
|
|
2019-04-14
|
100
|
100
|
100
|
100
|
100
|
100
|
|
2019-04-15
|
100
|
100
|
100
|
100
|
100
|
100
|
|
Target Availability for each site is 97.0%
|
Red <90%
|
Orange <97%
|
Day |
Atlas HC |
CMS HC |
Comment
|
2019-03-03 |
100 |
94 |
|
2019-03-04 |
100 |
98 |
|
2019-03-05 |
100 |
99 |
|
2019-03-06 |
100 |
100 |
|
2019-03-07 |
100 |
100 |
|
2019-03-08 |
100 |
100 |
|
2019-04-09 |
100 |
- |
|
Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud