Difference between revisions of "Tier1 Operations Report 2015-04-15"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(12 intermediate revisions by 2 users not shown)
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 8th to 15th April 2015.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 8th to 15th April 2015.
 
|}
 
|}
−
* On Wednesday 1st April there was a problem with the Argus service. The service was restarted before there were any tickets.
+
* On Wednesday 8th April a minor network change triggered a wider problem across the Tier1 network and caused significant disruption of services. The planned Castor upgrade for that day had to be abandoned. A post mortem is being generated for this incident.
−
* On Thursday 2nd April, a HyperVisor locked up and this caused a production squid and a test fts machine to be unavailable for some hours.  
+
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 33: Line 32:
 
|}
 
|}
 
* We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
 
* We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
 +
 +
* The bypass link from the Tier1 to the UKLR has been restored to 2 x 10Gb/s but traffic going outbound is not being correctly balanced across the two links.  This is still under investigation.
 +
 +
* Since 27th March there has been sporadic low level packet loss on the OPN primary link to CERN.  This is being investigated. 
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 54: Line 57:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
 +
* Castor was successfully upgraded to version 2.1.14-15 this morning.
 
* gfal2 and davix rpms are in the process of being updated across the worker nodes.
 
* gfal2 and davix rpms are in the process of being updated across the worker nodes.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
Line 104: Line 108:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
−
* Separate some non-Tier1 services off our network so as to be able to more easily investigate the router problems.
+
* Turn off CREAM CEs (planned for 5th May).
 +
* Turn off ARC-CE05. This will leave four ARC CEs (as planned.) This fifth was set-up to temporarily workround a specific problem and is no longer required.
 
'''Listing by category:'''
 
'''Listing by category:'''
 
* Databases:
 
* Databases:
 
** Switch LFC/3D to new Database Infrastructure.
 
** Switch LFC/3D to new Database Infrastructure.
−
** Update to Oracle 11.2.0.4
+
** Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases (Castor, LFC and Atlas Frontier)
 
* Castor:
 
* Castor:
 
** Update SRMs to new version (includes updating to SL6).
 
** Update SRMs to new version (includes updating to SL6).
 
** Fix discrepancies found in some of the Castor database tables and columns. (The issue has no operational impact.)
 
** Fix discrepancies found in some of the Castor database tables and columns. (The issue has no operational impact.)
 
* Networking:
 
* Networking:
 +
** Separate some non-Tier1 services off our network so as to be able to more easily investigate the router problems.
 
** Resolve problems with primary Tier1 Router
 
** Resolve problems with primary Tier1 Router
 
** Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
 
** Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
Line 194: Line 200:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 +
|-
 +
| 113010
 +
| Green
 +
| Very Urgent
 +
| Waiting Reply
 +
| 2015-04-13
 +
| 2015-04-14
 +
| SNO+
 +
| Job statuses reported by WMS not updating
 +
|-
 +
| 112977
 +
| Green
 +
| Very Urgent
 +
| In Progress
 +
| 2015-04-10
 +
| 2015-04-13
 +
| CMS
 +
| High failure rate for RAL!
 +
|-
 +
| 112896
 +
| Green
 +
| Urgent
 +
| In Progress
 +
| 2015-04-09
 +
| 2015-04-09
 +
| CMS
 +
| Please check this dataset
 
|-
 
|-
 
| 112866
 
| 112866

Latest revision as of 13:27, 15 April 2015

RAL Tier1 Operations Report for 15th April 2015

Review of Issues during the week 8th to 15th April 2015.
  • On Wednesday 8th April a minor network change triggered a wider problem across the Tier1 network and caused significant disruption of services. The planned Castor upgrade for that day had to be abandoned. A post mortem is being generated for this incident.
Resolved Disk Server Issues
  • None
Current operational status and issues
  • We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
  • The bypass link from the Tier1 to the UKLR has been restored to 2 x 10Gb/s but traffic going outbound is not being correctly balanced across the two links. This is still under investigation.
  • Since 27th March there has been sporadic low level packet loss on the OPN primary link to CERN. This is being investigated.
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • Castor was successfully upgraded to version 2.1.14-15 this morning.
  • gfal2 and davix rpms are in the process of being updated across the worker nodes.
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
cream-ce01.gridpp.rl.ac.uk, cream-ce02.gridpp.rl.ac.uk SCHEDULED OUTAGE 05/05/2015 12:00 02/06/2015 12:00 28 days, Decommissioning of CREAM CEs (cream-ce01.gridpp.rl.ac.uk, cream-ce02.gridpp.rl.ac.uk).
All Castor (All SRMs) SCHEDULED OUTAGE 15/04/2015 09:30 15/04/2015 15:30 6 hours Castor upgrade to 2.1.14-15 as this was postponed last week.
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Turn off CREAM CEs (planned for 5th May).
  • Turn off ARC-CE05. This will leave four ARC CEs (as planned.) This fifth was set-up to temporarily workround a specific problem and is no longer required.

Listing by category:

  • Databases:
    • Switch LFC/3D to new Database Infrastructure.
    • Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases (Castor, LFC and Atlas Frontier)
  • Castor:
    • Update SRMs to new version (includes updating to SL6).
    • Fix discrepancies found in some of the Castor database tables and columns. (The issue has no operational impact.)
  • Networking:
    • Separate some non-Tier1 services off our network so as to be able to more easily investigate the router problems.
    • Resolve problems with primary Tier1 Router
    • Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
    • Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
    • Make routing changes to allow the removal of the UKLight Router.
  • Fabric
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report.
Service Scheduled? Outage/At Risk Start End Duration Reason
All Castor (All SRMs) SCHEDULED OUTAGE 15/04/2015 09:30 15/04/2015 15:30 6 hours Castor upgrade to 2.1.14-15 as this was postponed last week.
Whole Site UNSCHEDULED WARNING 08/04/2015 18:57 09/04/2015 11:00 16 hours and 3 minutes Network problems continue at RAL
Whole Site UNSCHEDULED OUTAGE 08/04/2015 18:07 08/04/2015 19:06 59 minutes Network problems continue at RAL
Whole Site UNSCHEDULED OUTAGE 08/04/2015 10:31 08/04/2015 18:03 7 hours and 32 minutes Networking problem at RAL, we are investigating
All Castor (All SRMs) SCHEDULED OUTAGE 08/04/2015 10:00 08/04/2015 14:00 4 hours Upgrade of Castor storage to version 2.1.14-15
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
113010 Green Very Urgent Waiting Reply 2015-04-13 2015-04-14 SNO+ Job statuses reported by WMS not updating
112977 Green Very Urgent In Progress 2015-04-10 2015-04-13 CMS High failure rate for RAL!
112896 Green Urgent In Progress 2015-04-09 2015-04-09 CMS Please check this dataset
112866 Green Less Urgent In Progress 2015-04-02 2015-04-07 CMS Many jobs are failed/aborted at T1_UK_RAL
112819 Green Less Urgent In Progress 2015-04-02 2015-04-07 SNO+ ArcSync hanging
112721 Green Less Urgent waiting for reply 2015-03-28 2015-04-06 Atlas RAL-LCG2: SOURCE Failed to get source file size
112713 Green Urgent In Progress 2015-03-27 2015-03-31 CMS Please clean up unmerged area - RAL
111699 Yellow Less Urgent In Progress 2015-02-10 2015-03-23 Atlas gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
109694 Red Urgent In Progress 2014-11-03 2015-03-31 SNO+ gfal-copy failing for files at RAL
108944 Red Less Urgent In Progress 2014-10-01 2015-03-30 CMS AAA access test failing at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
08/04/15 70.5 100 65.0 65.0 62.0 76 77 Networking problems before, and at start of, Castor upgrade (which was pulled).
09/04/15 100 100 100 100 100 100 97
10/04/15 100 100 100 100 100 100 84
11/04/15 100 100 100 100 100 100 95
12/04/15 100 100 100 100 100 100 97
13/04/15 100 100 100 100 100 100 100
14/04/15 100 100 100 100 100 100 100
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2015-04-15&oldid=8190"