RAL Tier1 Operations Report for 5th April2017
| Review of Issues during the week 29th March to 5th April 2017.
|
- The Upgrade of the LHCb SRMs last Thursday (23rd March) went well. However, there were problems upgrading the Atlas ones yesterday (Tuesday 28th). This problem is now understood and the upgrade is continuing today.
- Some batch job submission errors have been seen by CMS and LHCb. These are not yet understood.
- Over the weekend there were two instances where a number of Tier1 systems lost connections between themselves. This was transitory and connection re-established themselves. The cause of this is not yet understood.
| Resolved Disk Server Issues
|
- GDSS647 (LHCbDst - D1T0) crashed in the early hours of Tuesday 28th March. The RAID card had become confused following the failure of a disk drive. It was being drained and was returned to service this morning (29th Mar).
| Current operational status and issues
|
- We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities but the level of failures is reduced as compared to a few weeks ago.
| Ongoing Disk Server Issues
|
- GDSS780 (LHCbDst - D1T0) had failed on Sunday (26th) and was put back in service the following day. However, it crashed again this morning and is currently under investigation.
| Limits on concurrent batch system jobs.
|
- Atlas Pilot (Analysis) 1500
- CMS Multicore 460
| Notable Changes made since the last meeting.
|
- SRM Upgrades (to version version 2.1.16-10 and the OS to SL6) were carried out for LHCb last Thursday (23rd March) and are ongoing for Atlas. CMS and 'GEN' instances planned for tomorrow.
- Work ongoing on replacing two of the chillers. The first replacement has been commissioned. The second one is due to be shipped out tomorrow (Thursday 30th March).
- This morning (Wed 29th March) the web alias www.gridpp.rl.ac.uk was flipped to point to a new web server.
- Out of Hours cover for the CEPH ECHO service is being piloted.
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
Pending - but not yet formally announced:
- Update Castor SRMs - ongoing.
- Chiller replacement - work ongoing.
- Merge AtlasScratchDisk into larger Atlas disk pool.
Listing by category:
- Castor:
- Update SRMs to new version, including updating to SL6.
- Bring some newer disk servers ('14 generation) into service, replacing some older ('12 generation) servers.
- Networking
- Enable first services on production network with IPv6 once addressing scheme agreed.
- Infrastructure:
- Two of the chillers supplying the air-conditioning for the R89 machine room are being replaced.
| Entries in GOC DB starting since the last report.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| srm-lhcb.gridpp.rl.ac.uk,
|
UNSCHEDULED
|
OUTAGE
|
31/03/2017 16:00
|
01/04/2017 10:43
|
18 hours and 43 minutes
|
There are problems with the LHCb Castor instance.
|
| srm-atlas.gridpp.rl.ac.uk
|
UNSCHEDULED
|
WARNING
|
29/03/2017 09:00
|
29/03/2017 14:12
|
5 hours and 12 minutes
|
upgrading Atlas srm software to 2.1.16
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 127388
|
Green
|
Less urgent
|
In Progress
|
2017-03-29
|
2017-04-03
|
LHCb
|
[FATAL] Connection error for some file
|
| 127240
|
Green
|
Urgent
|
In Progress
|
2017-03-21
|
2017-03-27
|
CMS
|
Staging Test at UK_RAL for Run2
|
| 126905
|
Green
|
Less Urgent
|
Waiting Reply
|
2017-03-02
|
2017-04-03
|
solid
|
finish commissioning cvmfs server for solidexperiment.org
|
| 126184
|
Amber
|
Less Urgent
|
In Progress
|
2017-01-26
|
2017-02-07
|
Atlas
|
Request of inputs for new sites monitoring
|
| 124876
|
Red
|
Less Urgent
|
On Hold
|
2016-11-07
|
2017-01-01
|
OPS
|
[Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
|
| 117683
|
Red
|
Less Urgent
|
On Hold
|
2015-11-18
|
2017-03-02
|
|
CASTOR at RAL not publishing GLUE 2.
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC ECHO = Atlas ECHO (Template 841);CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
Atlas HC ECHO |
CMS HC |
Comment
|
| 29/03/17 |
100 |
100 |
100 |
97 |
83 |
100 |
99 |
100 |
SRM test failures
|
| 30/03/17 |
100 |
100 |
100 |
100 |
88 |
100 |
100 |
100 |
SRM test failures
|
| 31/03/17 |
100 |
100 |
100 |
98 |
50 |
96 |
100 |
100 |
SRM test failures
|
| 01/04/17 |
100 |
100 |
67 |
100 |
79 |
100 |
100 |
100 |
Atlas: Missing data; LHCb: SRM test failures
|
| 02/034/17 |
100 |
100 |
100 |
98 |
100 |
100 |
89 |
100 |
SRM test failures
|
| 03/04/17 |
100 |
100 |
100 |
97 |
82 |
98 |
94 |
99 |
SRM test failures
|
| 04/04/17 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
