RAL Tier1 Operations Report for 12th July 2017
| Review of Issues during the week 5th to 12th July 2017.
|
- There is a power outage scheduled for the Atlas Building (R26) over the weekend of 29/30 July. Preparations are being made to remove any impact this may have on operational services.
- There were problems with the Atlas SRMs both on Saturday 8th July and again during the evening of Monday 10th July. Some of the SRM systems were found to be unresponsive. These problems were resolved by the out of hours on-call team.
- One of the CEPH gateway systems has had a hardware fault and was taken out of use (and out of the DNS alias) between the 6th and 11th July. There were elevated levels of gridftp transfer failures while the machine was removed, with errors indicating that the gateways are unable to serve more traffic (with the current connection limits.
- Two of the OCF '14 disk servers (One in each of AtlasDataDisk and CMSDisk) were removed from production for a couple of hours on Monday morning (10th July) for RAID card firmware updates. These were done ahead of the updating of the remaining systems in this batch planned for next week.
| Resolved Disk Server Issues
|
| Current operational status and issues
|
- We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities. CMS are also looking at file access performance and have turned off "lazy-download". The CMS SRM SAM test success rate has improved since the Castor 2.1.16 upgrade on the 25th May, although is still not 100%. It is still planned to re-visit this issue now Castor has been upgraded.
- There is a problem on the site firewall which is causing problems for some specific data flows. Discussions have been taking place with the vendor. It is expected that this is having an effect on our data that flows through the firewall (such as to/from worker nodes).
| Ongoing Disk Server Issues
|
| Limits on concurrent batch system jobs.
|
| Notable Changes made since the last meeting.
|
- The number of placement groups in the Echo CEPH Atlas pool contiues to be increased. This is in preparation for the increase in storage capacity when new hardware is added.
- CVMFS Stratum 0 and 1 nodes updated to cvmfs-server version v2.3.5-1.
- Security patching being carried out across systems.
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
Pending - but not yet formally announced:
- Firmware updates in OCF 14 disk servers. These will be done next week (17-21 July).
- Upgrade the FTS3 service to a version that will no longer support the SOAP interface. The SOAP interface will be disabled on the 17th July.
- Increase the number of placement groups in the Atlas Echo CEPH pool. (Ongoing)
Listing by category:
- Castor:
- Move to generic Castor headnodes.
- Merge AtlasScratchDisk into larger Atlas disk pool.
- Echo:
- Increase the number of placement groups in the Atlas Echo CEPH pool.
- Networking
- Enable first services on production network with IPv6 now that the addressing scheme has been agreed. (Perfsonar already working over IPv6).
- Services
- The production FTS needs updating. This will no longer support the SOAP interface. (The "test" FTS , used by Atlas, has already been upgraded.)
| Entries in GOC DB starting since the last report.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 129342
|
Green
|
Urgent
|
In Progress
|
2017-07-04
|
2017-07-10
|
|
[Rod Dashboard] Issue detected : org.sam.SRM-Put-ops@srm-mice.gridpp.rl.ac.uk
|
| 129059
|
Yellow
|
Very Urgent
|
Waiting for Reply
|
2017-06-20
|
2017-06-28
|
LHCb
|
Timeouts on RAL Storage
|
| 128991
|
Green
|
Less Urgent
|
In Progress
|
2017-06-16
|
2017-07-05
|
Solid
|
solidexperiment.org CASTOR tape support
|
| 127597
|
Red
|
Urgent
|
On Hold
|
2017-04-07
|
2017-06-14
|
CMS
|
Check networking and xrootd RAL-CERN performance
|
| 124876
|
Red
|
Less Urgent
|
On Hold
|
2016-11-07
|
2017-01-01
|
OPS
|
[Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
|
| 117683
|
Red
|
Less Urgent
|
On Hold
|
2015-11-18
|
2017-07-06
|
|
CASTOR at RAL not publishing GLUE 2.
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC Echo = Atlas Echo (Template 841);CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas Echo |
Atlas HC |
Atlas HC Echo |
CMS HC |
Comment
|
| 05/07/17 |
100 |
100 |
100 |
98 |
100 |
100 |
100 |
100 |
100 |
Few SRM test failures (mostly User timeout)
|
| 06/07/17 |
100 |
100 |
100 |
97 |
100 |
100 |
97 |
100 |
100 |
Few SRM test failures (User timeout)
|
| 07/07/17 |
100 |
100 |
100 |
99 |
100 |
100 |
100 |
99 |
100 |
Single SRM test failure (User timeout)
|
| 08/07/17 |
100 |
100 |
87 |
89 |
100 |
100 |
91 |
100 |
99 |
Atlas: Problem on two of the four SRMs. On-Call team intervened. CMS: mix of SRM test failures.
|
| 09/07/17 |
100 |
100 |
100 |
92 |
100 |
100 |
100 |
100 |
99 |
SRM test failures
|
| 10/07/17 |
100 |
100 |
83 |
95 |
92 |
100 |
75 |
100 |
99 |
Atlas: Failures on the SRM SAM test during evening. On-call team intervened. CMS: Block of failed tests for the CEs and sporadic SRM test failures. LHCb: Some SRM test failures ('Communication error’ and ‘SRM_FILE_BUSY‘)
|
| 11/07/17 |
100 |
100 |
100 |
97 |
96 |
100 |
100 |
100 |
100 |
CMS: SRM test failures. LHCb: Single SRM test failure: could not open connection to srm-lhcb.gridpp.rl.ac.uk:8443
|
|
