RAL Tier1 Operations Report for 25th May 2016
Review of Issues during the week 18th to 25th May 2016.
|
- There were problems with the Tier1 Tape Library from Monday (9th) through to Friday (13th). In the week beforehand one of the two "elevators" in the tape library had failed. This was not an operational problem. However, on the Monday the second 'elevator' stopped working too. This led to some tape mounts failing. At the end of the afternoon the following day (Tuesday) there was a more severe failure within the library and all tape mounts stopped. While the vendor was working on the problem some reconfiguration was done to make use of a second tape library and from Thursday 12th we were able to write Tier1 data. The vendor fully fixed the problem on Friday (13th) and during that afternoon full tape access (read & write) was restored. The tape system was left "at risk" over the weekend.
Resolved Disk Server Issues
|
Current operational status and issues
|
- There is a problem seen by LHCb of a low but persistent rate of failure when copying the results of batch jobs to Castor. There is also a further problem that sometimes occurs when these (failed) writes are attempted to storage at other sites. A recent modification has improved, but not completed fixed this.
- The intermittent, low-level, load-related packet loss seen over external connections is still being tracked. Likewise we have been working to understand some remaining low level of packet loss seen within a part of our Tier1 network.
Ongoing Disk Server Issues
|
- The first two disk servers listed below have failed more than once and are being put through a week long acceptance test.
- GDSS635 (AtlasTape - D0T1) failed in the early hours of the 20th April.
- GDSS619 (GenTape - D0T1) failed on 26th April only a few hours after being put back into service after a previous failure.
- GDSS664 (AtlasTape - D0T1) failed on Monday morning, 16th May. A verification of the RAID array found a faulty disk which has been replaced. Expected back in production later today.
Notable Changes made since the last meeting.
|
- The migration of Atlas data from "C" to "D" tapes is continuing.
- Updated Condor nodes and UIs to condor version 8.4.6.
- All Tapeservers now running Castor version 2.1.16-0
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
lcgwms06.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
01/06/2016 11:00
|
30/06/2016 11:00
|
29 days,
|
Server lcgwms06.gridpp.rl.ac.uk Decommissioning
|
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
- The Castor 2.1.15 update is pending. Testing has shown a database related performance issue which is being followed up. We await successful resolution of that problem and completion of testing before scheduling. Following advice from the developers we will not upgrade the SRMs before the Castor 2.1.15 upgrade.
- Decommissioning of "GEN Scratch" storage in Castor. (Formally announced by EGI broadcast).
- Decommissioning of lcgwms06. This will leave two WMS systems remaining in service.
- The HAProxy Load Balancer will be added in front of the Production FTS3 Service.
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Update SRMs to new version (includes updating to SL6).
- Update to Castor version 2.1.15.
- Migration of data from T10KC to T10KD tapes (Affects Atlas & LHCb data).
- Networking:
- Replace the UKLight Router. Then upgrade the 'bypass' link to the RAL border routers to 2*40Gbit.
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, LFC)
- Grid Services
- Once the use of the Load Balancer (HAProxy) has been proven for the test FTS service it will be extended to other services.
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
srm-lhcb.gridpp.rl.ac.uk,
|
UNSCHEDULED
|
WARNING
|
13/05/2016 15:30
|
16/05/2016 11:15
|
2 days, 19 hours and 45 minutes
|
At-risk on the tape robots after fixing the problem with the robotics.
|
srm-lhcb.gridpp.rl.ac.uk,
|
UNSCHEDULED
|
OUTAGE
|
12/05/2016 19:00
|
13/05/2016 18:00
|
23 hours
|
Tape Library not available. No tape access.
|
All Castor SRMs with tape access.
|
UNSCHEDULED
|
WARNING
|
12/05/2016 19:00
|
16/05/2016 11:16
|
3 days, 16 hours and 16 minutes
|
Tape Library not available. No tape access.
|
srm-lhcb.gridpp.rl.ac.uk,
|
UNSCHEDULED
|
OUTAGE
|
11/05/2016 19:00
|
12/05/2016 19:00
|
24 hours
|
Tape Library not available. No tape access.
|
All Castor SRMs with tape access.
|
UNSCHEDULED
|
WARNING
|
11/05/2016 19:00
|
12/05/2016 19:00
|
24 hours
|
Tape Library not available. No tape access.
|
All Castor SRMs with tape access.
|
UNSCHEDULED
|
WARNING
|
10/05/2016 19:00
|
11/05/2016 19:00
|
24 hours
|
Problem still ongoing with tape robot affecting some tape mounts. (Engineer returning tomorrow)
|
Open GGUS Tickets (Snapshot during morning of meeting)
|
GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
121322
|
Green
|
Less Urgent
|
In Progress
|
2016-05-10
|
2016-05-10
|
SNO+
|
Unable to access grid storage
|
120954
|
Green
|
Less Urgent
|
In Progress
|
2016-04-21
|
2016-05-11
|
LHCb
|
SRM endpoint entry in GOCDB
|
120920
|
Green
|
Less Urgent
|
In Progress
|
2016-04-19
|
2016-05-06
|
SNO+
|
XRootD issues at RAL
|
120810
|
Green
|
Urgent
|
On Hold
|
2016-04-13
|
2016-04-27
|
Biomed
|
Decommissioning of SE srm-biomed.gridpp.rl.ac.uk - forbid write access for biomed users
|
120350
|
Green
|
Less Urgent
|
In Progress
|
2016-03-22
|
2016-05-06
|
LSST
|
Enable LSST at RAL
|
119841
|
Amber
|
Less Urgent
|
On Hold
|
2016-03-01
|
2016-04-26
|
LHCb
|
HTTP support for lcgcadm04.gridpp.rl.ac.uk
|
117683
|
Yellow
|
Less Urgent
|
On Hold
|
2015-11-18
|
2016-04-05
|
|
CASTOR at RAL not publishing GLUE 2
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 729); CMS HC = CMS HammerCloud
Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
18/05/16 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
19/05/16 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
20/05/16 |
100 |
100 |
100 |
97 |
100 |
86 |
76 |
Problem with test on all CEs. Looks like a CMS problem.
|
21/05/16 |
100 |
100 |
28 |
98 |
100 |
99 |
90 |
Atlas: Central monitoring problem; CMS: Single SRM test failure ("Input/output error")
|
22/05/16 |
100 |
100 |
76 |
100 |
100 |
100 |
94 |
Atlas: Central monitoring problem
|
23/05/16 |
100 |
100 |
100 |
98 |
100 |
100 |
100 |
Single SRM Test failure: User timeout.
|
24/05/16 |
100 |
100 |
100 |
100 |
96 |
98 |
100 |
Single SRM Test failure: (No such file or directory)
|