Difference between revisions of "Tier1 Operations Report 2017-09-06"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(10 intermediate revisions by one user not shown)
Line 10: Line 10:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 30th August to 6th September 2017.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 30th August to 6th September 2017.
 
|}
 
|}
−
* There have been problems with Echo since last weekend. Although now resolved there was a loss of around 22,000 Atlas files (of which 3285 files were unique). One CEPH "placement group" was lost. This is most likely due to some low level disk errors not being handled cleanly. While the problem was ongoing it also revealed a problem with the Echo gateways (including those on the worker nodes) which created many threads - blocking activity.
+
* There were further problems with Echo at the end of last week. These appear similar to those of the previous week but we believe are now better understood. Media errors in the hardware that has been added into Echo exposed a bug in the CEPH Erasure Coding when back-filling. Furthermore a problem in the Echo gateways when a CEPH placement group is unavailable (as in this case) requires that the gateways be stopped while the problem is fixed.
−
* Four files were reported lost to CMS from a tape. During a CMS mass recall the monitoring noted that one tape was generating errors and causing tape drives to go down. The tape was put into repack.  After various attempts at repacking and reading files direct from tape, there were four files left unreadable.  
+
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 32: Line 31:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
* We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities. CMS are also looking at file access performance and have turned off "lazy-download". The CMS SRM SAM test success rate has improved since the Castor 2.1.16 upgrade on the 25th May, although is still not 100%. Our investigations are ongoing.
+
* We were seeing a high rate of failures of the CMS SAM tests against the SRM which affected our (CMS) availabilities. The last few weeks have shown significantly better availability test results - although it is not understood why. We continue to monitor this.
 
* There is a problem on the site firewall which is causing problems for some specific data flows. Discussions have been taking place with the vendor. This is having an effect on our data that flows through the firewall (such as to/from worker nodes).
 
* There is a problem on the site firewall which is causing problems for some specific data flows. Discussions have been taking place with the vendor. This is having an effect on our data that flows through the firewall (such as to/from worker nodes).
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
Line 67: Line 66:
 
|}
 
|}
 
* Further services migrated from our old Windows HyperV2008 infrastructure to hypervisors running HyperV2012 as we prepare to fully decommission the HyperV2008 systems.
 
* Further services migrated from our old Windows HyperV2008 infrastructure to hypervisors running HyperV2012 as we prepare to fully decommission the HyperV2008 systems.
 +
* On the 30th August the site firewall was upgrade to a newer more supported version of software.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 77: Line 77:
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
|}
 
|}
−
{| border=1 align=center
+
* None
−
|- bgcolor="#7c8aaf"
+
−
! Service
+
−
! Scheduled?
+
−
! Outage/At Risk
+
−
! Start
+
−
! End
+
−
! Duration
+
−
! Reason
+
−
|-
+
−
| srm-superb.gridpp.rl.ac.uk,
+
−
| SCHEDULED
+
−
| OUTAGE
+
−
| 20/07/2017 16:00
+
−
| 30/08/2017 13:00
+
−
| 40 days, 21 hours
+
−
| SuperB no longer supported on Castor storage. Retiring endpoint.
+
−
|-
+
−
| srm-hone.gridpp.rl.ac.uk,
+
−
| SCHEDULED
+
−
| OUTAGE
+
−
| 20/07/2017 16:00
+
−
| 30/08/2017 13:00
+
−
| 40 days, 21 hours
+
−
| H1 no longer supported on Castor storage. Retiring endpoint.
+
−
|}
+
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 148: Line 123:
 
! Reason
 
! Reason
 
|-
 
|-
−
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk,
+
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk
 
| UNSCHEDULED
 
| UNSCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 25/08/2017 12:00
+
| 01/09/2017 12:00
−
| 25/08/2017 15:00
+
| 01/09/2017 17:00
−
| 3 hours
+
| 5 hours
−
| continued problems with Echo instance at RAL
+
| Problems with Ceph/Echo instance at RAL
 
|-
 
|-
−
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk,
+
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk
 
| UNSCHEDULED
 
| UNSCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 24/08/2017 11:23
+
| 31/08/2017 19:30
−
| 25/08/2017 12:00
+
| 01/09/2017 13:00
−
| 1 day, 37 minutes
+
| 17 hours and 30 minutes
−
| Ceph team need to take the Echo service offline while dealing with ongoing issues
+
| Problems with Ceph, service is degraded
 
|-
 
|-
−
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk,
+
| srm-superb.gridpp.rl.ac.uk
−
| UNSCHEDULED
+
−
| WARNING
+
−
| 23/08/2017 11:00
+
−
| 25/08/2017 16:00
+
−
| 2 days, 5 hours
+
−
| Ongoing problems with Ceph, service is still degraded
+
−
|-
+
−
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk,
+
−
| UNSCHEDULED
+
−
| WARNING
+
−
| 22/08/2017 10:03
+
−
| 23/08/2017 10:00
+
−
| 23 hours and 57 minutes
+
−
| Ongoing problems with Ceph, service is degraded
+
−
|-
+
−
| srm-hone.gridpp.rl.ac.uk,
+
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
Line 186: Line 145:
 
| 30/08/2017 13:00
 
| 30/08/2017 13:00
 
| 40 days, 21 hours
 
| 40 days, 21 hours
−
| H1 no longer supported on Castor storage. Retiring endpoint.
+
| SuperB no longer supported on Castor storage. Retiring endpoint.
 
|-
 
|-
−
| srm-superb.gridpp.rl.ac.uk,
+
| srm-hone.gridpp.rl.ac.uk
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
Line 194: Line 153:
 
| 30/08/2017 13:00
 
| 30/08/2017 13:00
 
| 40 days, 21 hours
 
| 40 days, 21 hours
−
| SuperB no longer supported on Castor storage. Retiring endpoint.
+
| H1 no longer supported on Castor storage. Retiring endpoint.
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 214: Line 173:
 
| Green
 
| Green
 
| Urgent
 
| Urgent
−
| In Progress
+
| Waiting for Reply
 
| 2017-08-24
 
| 2017-08-24
−
| 2017-08-25
+
| 2017-08-31
 
| MICE
 
| MICE
 
| Timeouts when copying MICE reco data to CASTOR
 
| Timeouts when copying MICE reco data to CASTOR
−
|-
 
−
| 130197
 
−
| Green
 
−
| Urgent
 
−
| Waiting for Reply
 
−
| 2017-08-23
 
−
| 2017-08-24
 
−
| SNO+
 
−
| LFC registration failing
 
 
|-
 
|-
 
| 130193
 
| 130193
Line 234: Line 184:
 
| In Progress
 
| In Progress
 
| 2017-08-23
 
| 2017-08-23
−
| 2017-08-24
+
| 2017-09-04
 
| CMS
 
| CMS
 
| Staging from RAL tape systems
 
| Staging from RAL tape systems
Line 293: Line 243:
 
| 31/08/17 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
| 31/08/17 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
|-
 
|-
−
| 01/09/17 || style="background-color: lightgrey;" | 97.9 || 100 || 100 || 100 || 100 || 100 || CE test failures.
+
| 01/09/17 || style="background-color: lightgrey;" | 97.9 || 100 || 100 || 100 || 100 || 100 || CE test failures. (Possibly triggered by problems on Echo).
 
|-
 
|-
 
| 02/09/17 || 100 || 100 || 100 || 100 || 100 || 100 ||  
 
| 02/09/17 || 100 || 100 || 100 || 100 || 100 || 100 ||  
Line 343: Line 293:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
|}
 
|}
−
* None yet
+
* There has been a successful test transfer of CMS data into Echo using PhEDEx.

Latest revision as of 13:38, 6 September 2017

RAL Tier1 Operations Report for 6th September 2017

Review of Issues during the week 30th August to 6th September 2017.
  • There were further problems with Echo at the end of last week. These appear similar to those of the previous week but we believe are now better understood. Media errors in the hardware that has been added into Echo exposed a bug in the CEPH Erasure Coding when back-filling. Furthermore a problem in the Echo gateways when a CEPH placement group is unavailable (as in this case) requires that the gateways be stopped while the problem is fixed.
Resolved Disk Server Issues
  • GDSS746 (AtlasDataDisk - D1T0) was unavailable for around six hours yesterday (Tuesday 5th Sep) after a replacement drive for a fault also failed.
Current operational status and issues
  • We were seeing a high rate of failures of the CMS SAM tests against the SRM which affected our (CMS) availabilities. The last few weeks have shown significantly better availability test results - although it is not understood why. We continue to monitor this.
  • There is a problem on the site firewall which is causing problems for some specific data flows. Discussions have been taking place with the vendor. This is having an effect on our data that flows through the firewall (such as to/from worker nodes).
Ongoing Disk Server Issues
  • None
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • Further services migrated from our old Windows HyperV2008 infrastructure to hypervisors running HyperV2012 as we prepare to fully decommission the HyperV2008 systems.
  • On the 30th August the site firewall was upgrade to a newer more supported version of software.
Declared in the GOC DB
  • None
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Pending - but not yet formally announced:

  • Upgrade the FTS3 service to a version that will no longer support the SOAP interface. The SOAP interface was disabled on Monday (17th July).
  • Re-distribute the data in Echo onto the 2015 capacity hardware. (Ongoing)

Listing by category:

  • Castor:
    • Move to generic Castor headnodes.
  • Echo:
    • Re-distribute the data in Echo onto the remaining 2015 capacity hardware.
  • Networking
    • Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, all squids and the CVMFS Stratum-1 servers).
  • Services
    • The production FTS will be updated now the requirement to support the deprecated SOAP interface has been removed.
Entries in GOC DB starting since the last report.
Service Scheduled? Outage/At Risk Start End Duration Reason
gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk UNSCHEDULED OUTAGE 01/09/2017 12:00 01/09/2017 17:00 5 hours Problems with Ceph/Echo instance at RAL
gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk UNSCHEDULED OUTAGE 31/08/2017 19:30 01/09/2017 13:00 17 hours and 30 minutes Problems with Ceph, service is degraded
srm-superb.gridpp.rl.ac.uk SCHEDULED OUTAGE 20/07/2017 16:00 30/08/2017 13:00 40 days, 21 hours SuperB no longer supported on Castor storage. Retiring endpoint.
srm-hone.gridpp.rl.ac.uk SCHEDULED OUTAGE 20/07/2017 16:00 30/08/2017 13:00 40 days, 21 hours H1 no longer supported on Castor storage. Retiring endpoint.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
130207 Green Urgent Waiting for Reply 2017-08-24 2017-08-31 MICE Timeouts when copying MICE reco data to CASTOR
130193 Green Urgent In Progress 2017-08-23 2017-09-04 CMS Staging from RAL tape systems
128991 Green Less Urgent On Hold 2017-06-16 2017-08-24 Solid solidexperiment.org CASTOR tape support
127597 Red Urgent On Hold 2017-04-07 2017-06-14 CMS Check networking and xrootd RAL-CERN performance
124876 Red Less Urgent On Hold 2016-11-07 2017-01-01 OPS [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
117683 Red Less Urgent On Hold 2015-11-18 2017-07-06 CASTOR at RAL not publishing GLUE 2.
Availability Report
Day OPS Alice Atlas CMS LHCb Atlas Echo Comment
30/08/17 100 100 100 100 100 100
31/08/17 100 100 100 100 100 100
01/09/17 97.9 100 100 100 100 100 CE test failures. (Possibly triggered by problems on Echo).
02/09/17 100 100 100 100 100 100
03/09/17 100 100 100 100 100 100
04/09/17 100 100 100 100 100 100
05/09/17 100 100 100 100 100 100
Hammercloud Test Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC Echo = Atlas Echo (Template 841);CMS HC = CMS HammerCloud

Day Atlas HC Atlas HC Echo CMS HC Comment
30/08/17 100 100 100
31/08/17 100 86 100
01/09/17 99 51 100
02/09/17 100 100 100
03/09/17 98 100 100
04/09/17 92 95 100
05/09/17 85 96 98
Notes from Meeting.
  • There has been a successful test transfer of CMS data into Echo using PhEDEx.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2017-09-06&oldid=15383"