Difference between revisions of "Tier1 Operations Report 2015-02-25"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
Line 67: Line 67:
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
|}
 
|}
−
{| border=1 align=center
+
None
−
|- bgcolor="#7c8aaf"
+
−
! Service
+
−
! Scheduled?
+
−
! Outage/At Risk
+
−
! Start
+
−
! End
+
−
! Duration
+
−
! Reason
+
−
|-
+
−
| Whole site
+
−
| SCHEDULED
+
−
| WARNING
+
−
| 19/02/2015 07:45
+
−
| 19/02/2015 08:30
+
−
| 45 minutes
+
−
| Warning during test of network router. We expect a 5 minute break in connectivity to site sometime during this window.
+
−
|}
+
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->

Revision as of 09:09, 25 February 2015

RAL Tier1 Operations Report for 25th February 2015

Review of Issues during the week 18th to 25th February 2015.
  • On Monday morning the Tier1 data link failed. This is the link between the Tier1 routers and the UKLight router). All data transfers stopped. The problem lasted for around an hour from 09:00.
  • A single file loss was reported to Atlas. This was picked up by the regular checksum valdiation process.
Resolved Disk Server Issues
  • GDSS656 (LHCbRawRdst - D1T0) failed on Thursday afternoon (12th Feb). It was returned to service on the Saturday morning (14th Feb). A disk drive failed while verifying the RAID array.
Current operational status and issues
  • We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • Condor update to version 8.2.7 applied to CEs and other internal Condor systems.
  • Test/first roll-out deployments of condor 8.2.7 and cvmfs-client 2.1.20 on Worker nodes.
  • Both production Atlas Frontier machines are now using Cronos as the Atlas 3D database backend instead of Ogma.
Declared in the GOC DB

None

Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Investigate problems on the primary Tier1 router. Discussions with the vendor are ongoing and a test is scheduled for tomorrow morning (19th Feb).
  • Physical move of 2011 disk server racks to make space for new delivery.

Listing by category:

  • Databases:
    • Switch LFC/3D to new Database Infrastructure.
    • Update to Oracle 11.2.0.4
  • Castor:
    • Update SRMs to new version (includes updating to SL6).
    • Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
    • Update Castor to 2.1-14-latest.
  • Networking:
    • Resolve problems with primary Tier1 Router
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
    • Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
  • Fabric
    • Physical move of 2011 disk server racks to make space for new delivery.
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report.
Service Scheduled? Outage/At Risk Start End Duration Reason
Many Castor services (srm-alice, srm-atlas, srm-biomed, srm-dteam, srm-ilc, srm-lhcb-tape, srm-lhcb, srm-mice, srm-na62, srm-pheno, srm-snoplus, srm-superb, srm-t2k SCHEDULED OUTAGE 24/02/2015 08:30 24/02/2015 17:06 8 hours and 36 minutes Outage of some parts of the Castor storage while disk servers are moved to make space for new equipment.
Whole site SCHEDULED WARNING 19/02/2015 07:45 19/02/2015 08:30 45 minutes Warning during test of network router. We expect a 5 minute break in connectivity to site sometime during this window.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
111699 Green Less Urgent Waiting Reply 2015-02-10 2015-02-18 Atlas gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
111120 Green Less Urgent Waiting Reply 2015-01-12 2015-01-22 Atlas large transfer errors from RAL-LCG2 to BNL-OSG2
109694 Red Urgent Waiting Reply 2014-11-03 2015-02-18 SNO+ gfal-copy failing for files at RAL
108944 Red Urgent In Progress 2014-10-01 2015-02-18 CMS AAA access test failing at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
18/02/15 100 100 100 100 100 100 100
19/02/15 100 100 98.0 100 100 98 97 Single SRM test failure
20/02/15 100 100 98.0 100 100 100 100 Single SRM test failure
21/02/15 100 100 100 100 100 100 99
22/02/15 100 100 98.0 100 100 100 99 Single SRM test failure
23/02/15 100 100 100 100 100 97 99
24/02/15 100 100 62.0 58.0 59.0 94 92 Planned move of disk server racks.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2015-02-25&oldid=7635"