RAL Tier1 Operations Report for 11th October 2017
| Review of Issues during the week 4th to 11th October 2017.
|
- There were problems with CMS Disk becoming full at the end of last week. The draining of older disk servers (2012 generation) - partly in anticipation of CMS moving to Echo had been taking place and this had taken CMS to below pledge. This was causing significant problems for CMS and on Friday five of this generation disk server were put back into service (a total of a bit over 400TB). (Although one disk server would have taken us well above pledge - just adding one would have resulting in hotspotting. Hence five available servers were added.) CMS SAM tests against the SRM were failing while CMSDisk was full.
| Resolved Disk Server Issues
|
- GDSS733 (LHCbDst - D1T0) crashed during the early evening of Saturday (7th Oct). It was put back in service around midday the following day read-only. The RAID array verification found one faulty disk drive that was replaced. However, another disk drive failed during the rebuild of the array after the first one was changed. That second disk drive has also been swapped out.
| Current operational status and issues
|
- There is a problem on the site firewall which is causing problems for some specific data flows. Discussions have been taking place with the vendor. This is having an effect on our data that flows through the firewall (such as to/from worker nodes).
| Ongoing Disk Server Issues
|
- GDSS788 (AtlasDataDisk - D1T0) crashed during the early evening of this morning (Wednesday 11th Oct). Investigations are ongoing.
| Limits on concurrent batch system jobs.
|
| Notable Changes made since the last meeting.
|
- Re-distribution of data in Echo onto the 2015 capacity hardware is ongoing.
- Update of the "Test" FTS3 (used by Atlas) service to version to 3.7.4 took place this morning.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgfts3.gridpp.rl.ac.uk
|
SCHEDULED
|
OUTAGE
|
12/10/2017 10:00
|
12/10/2017 13:00
|
3 hours
|
Upgrading FTS to 3.7.4
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
Pending - but not yet formally announced:
- Re-distribute the data in Echo onto the 2015 capacity hardware. (Ongoing)
Listing by category:
- Castor:
- Move to generic Castor headnodes.
- Echo:
- Re-distribute the data in Echo onto the remaining 2015 capacity hardware.
- Networking
- Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, all squids and the CVMFS Stratum-1 servers).
- Services
- The Production and "Test" (Atlas) FTS3 services will be merged and will make use of a resilient distributed database.
| Entries in GOC DB starting since the last report.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 130949
|
Green
|
Urgent
|
In Progress
|
2017-10-06
|
2017-10-10
|
CMS
|
Transfers failing to T1_UK_RAL_Disk
|
| 130883
|
Green
|
Urgent
|
In Progress
|
2017-10-02
|
2017-10-10
|
CMS
|
T1_UK_RAL_Disk to T2_US_Florida transfers are failing due to missing file(s)
|
| 130782
|
Green
|
Top Priority
|
In Progress
|
2017-09-27
|
2017-10-10
|
LHCb
|
Please deploy new version of HEP_OSlibs
|
| 130467
|
Green
|
Urgent
|
In Progress
|
2017-09-10
|
2017-09-25
|
CMS
|
T1_UK_RAL has SAM3 CE and SRM critical > 2hours
|
| 130207
|
Amber
|
Urgent
|
In Progress
|
2017-08-24
|
2017-09-06
|
MICE
|
Timeouts when copying MICE reco data to CASTOR
|
| 130193
|
Amber
|
Urgent
|
In Progress
|
2017-08-23
|
2017-09-29
|
CMS
|
Staging from RAL tape systems
|
| 127597
|
Red
|
Urgent
|
On Hold
|
2017-04-07
|
2017-10-05
|
CMS
|
Check networking and xrootd RAL-CERN performance
|
| 124876
|
Red
|
Less Urgent
|
On Hold
|
2016-11-07
|
2017-01-01
|
OPS
|
[Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
|
| 117683
|
Red
|
Less Urgent
|
On Hold
|
2015-11-18
|
2017-07-06
|
|
CASTOR at RAL not publishing GLUE 2.
|
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas Echo |
Comment
|
| 04/10/17 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 05/10/17 |
100 |
100 |
100 |
63 |
96 |
100 |
CMS failed SRM tests with Disk full; LHCb: Single SRM test failure.
|
| 06/10/17 |
100 |
100 |
100 |
43 |
100 |
100 |
CMS Disk full.
|
| 07/10/17 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 08/10/17 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 09/10/17 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 10/10/17 |
100 |
100 |
100 |
99 |
100 |
100 |
Problem with monitoring system.
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC Echo = Atlas Echo (Template 841);CMS HC = CMS HammerCloud
| Day |
Atlas HC |
Atlas HC Echo |
CMS HC |
Comment
|
| 04/10/17 |
90 |
87 |
100 |
|
| 05/10/17 |
99 |
100 |
100 |
|
| 06/10/17 |
98 |
100 |
100 |
|
| 07/10/17 |
96 |
100 |
100 |
|
| 08/10/17 |
100 |
100 |
100 |
|
| 09/10/17 |
99 |
100 |
100 |
|
| 10/10/17 |
97 |
98 |
99 |
|
- Once the FTS3 services have been merged and set-up on a resilient (distributed) database they will have IPv6 dual-stack enabled.
- Raja asked if the LHCb VO box could be made dual stack.
- Modified gateway code that uses less memory has been put on one (out of the five) Echo gateways. It will be rolled out across more of them in the next week or so.
- One file has been reported lost to LHCb. It was a unique copy.
