Difference between revisions of "Tier1 Operations Report 2016-11-23"

From GridPP Wiki
Jump to: navigation, search
(Created page with "==RAL Tier1 Operations Report for 23rd November 2016== __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Star...")
 
(→‎)
Line 106: Line 106:
 
! Duration
 
! Duration
 
! Reason
 
! Reason
 +
|-
 +
|gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk
 +
| UNSCHEDULED
 +
| OUTAGE
 +
| 18/11/2016 14:30
 +
| 21/11/2016 11:00
 +
| 2 days, 20 hours and 30 minutes
 +
| Echo cluster is coming back online, following on from shutdown for backend network upgrade behind Echo Storage service
 +
|-
 +
|gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk
 +
| UNSCHEDULED
 +
| OUTAGE
 +
| 16/11/2016 14:30
 +
| 18/11/2016 14:30
 +
| 2 day,
 +
| Follow on work dealing with bringing cluster back online after backend network upgrade behind Echo Storage service
 
|-
 
|-
 
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk
 
| gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk

Revision as of 11:39, 23 November 2016

RAL Tier1 Operations Report for 23rd November 2016

Review of Issues during the week 16th to 23rd November 2016.
  • There was a problem with one of the Atlas Frontier launchpads on Monday 14th. All three such systems were updated to the newer release (3.5.22)
Resolved Disk Server Issues
  • GDSS713 (CMS Disk - D1T0) crashed on Sunday morning. It was returned to service the following day - initially read-only. Our checks did not find any fault on the server.
Current operational status and issues
  • There is a problem seen by LHCb of a low but persistent rate of failure when copying the results of batch jobs to Castor. There is also a further problem that sometimes occurs when these (failed) writes are attempted to storage at other sites.
  • The intermittent, low-level, load-related packet loss that has been seen over external connections is still being tracked. The replacement of the UKLight router appears to have reduced this - but we are allowing more time to pass before drawing any conclusions.
Ongoing Disk Server Issues
  • None
Notable Changes made since the last meeting.
  • There was a short (one to two hour) interruption to tape mounts while the system that runs the tape library control software was swapped on Tuesday morning (15th Nov).
  • LHCb writing to the 'D' tapes. The migration of their data from 'C' to 'D' tapes is underway - with around 100 (some 10%) of the tapes done.
Declared in the GOC DB

None

Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Merge AtlasScratchDisk and LhcbUser into larger disk pools
    • Update to Castor version 2.1.15. Planning to roll out January 2017. (Proposed dates: 10th Jan: Nameserver; 17th Jan: First stager (LHCb); 24th Jan: Stager (Atlas); 26th Jan: Stager (GEN); 31st Jan: Final stager (CMS)).
    • Update SRMs to new version, including updating to SL6. This will be done after the Castor 2.1.15 update.
  • Fabric
    • Firmware updates on older disk servers.
Entries in GOC DB starting since the last report.
Service Scheduled? Outage/At Risk Start End Duration Reason
gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk UNSCHEDULED OUTAGE 18/11/2016 14:30 21/11/2016 11:00 2 days, 20 hours and 30 minutes Echo cluster is coming back online, following on from shutdown for backend network upgrade behind Echo Storage service
gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk UNSCHEDULED OUTAGE 16/11/2016 14:30 18/11/2016 14:30 2 day, Follow on work dealing with bringing cluster back online after backend network upgrade behind Echo Storage service
gridftp.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, s3.echo.stfc.ac.uk, xrootd.echo.stfc.ac.uk SCHEDULED OUTAGE 16/11/2016 10:30 16/11/2016 14:30 4 hours Upgrading backend network behind Echo Storage service
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
124876 Green Less Urgent On Hold 2016-11-07 2016-11-15 OPS [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
124785 Red Urgent Reopened 2016-11-02 2016-11-09 CMS Configuration updated AAA - CMS Site Name missing
124606 Yellow Very Urgent In Progress 2016-10-24 2016-11-01 CMS Consistency Check for T1_UK_RAL
122827 Green Less Urgent Waiting for Reply 2016-07-12 2016-10-11 SNO+ Disk area at RAL
120350 Yellow Less Urgent In Progress 2016-03-22 2016-08-09 LSST Enable LSST at RAL
117683 Red Less Urgent On Hold 2015-11-18 2016-10-05 CASTOR at RAL not publishing GLUE 2 (Updated. There are ongoing discussions with GLUE & WLCG)
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 808); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
16/11/16 100 100 100 100 100 99 N/A
17/11/16 100 100 100 100 100 100 100
18/11/16 100 100 100 100 100 100 100
19/11/16 100 100 100 100 100 100 100
20/11/16 100 100 100 100 100 100 100
21/11/16 100 100 100 100 100 100 100
22/11/16 100 100 100 100 100 100 100
Notes from Meeting.
  • None yet
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2016-11-23&oldid=12867"