Difference between revisions of "Tier1 Operations Report 2015-03-18"

From GridPP Wiki
Jump to: navigation, search
(Created page with "==RAL Tier1 Operations Report for 18th March 2015== __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Start Re...")
 
(→‎)
 
(14 intermediate revisions by one user not shown)
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 11th to 18th March 2015.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 11th to 18th March 2015.
 
|}
 
|}
−
* The planned move to put in a replacement primary router that was scehduled on the morning of the 5th March had to be delayed (to the morning of the 12th March).
+
* The replacement of the the primary Tier1 router did not fix the problem whereby the router fails after a few minutes in service.
−
* On Friday (6th March) around 15:00 there was a problem of high packet loss within the Tier1 network following a configuration error. This lasted around 20 minutes.
+
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 21: Line 20:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
−
* GDSS757 (CMSDisk - D1T0) has failed multiple times this year. It has now been drained and the vendor contacted for further diagnostics. The server did fail one more time during the drain.
+
* None
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 54: Line 53:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
−
* The roll-out of Condor version 8.2.7 and CVMFS version 2.1.20 to the batch farm is complete apart from mopping up a few nodes.
+
* Five additional disk servers have been deployed into Castor storage for ALICE_Disk. This roughly doubles the disk space available to ALICE which is now approximately 390TB.
−
* This morning (11th March) the firmware version in the pair of switches at the core of the Tier1 network was updated.
+
* On the CEs the upper memory limit was reduced from 5x requested memory to 3x.
−
* All data has now been migrated off the T10KA, as well as the T10KB tapes.
+
* ATLAS are now putting all UK transfers through our test FTS3.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 77: Line 76:
 
! Reason
 
! Reason
 
|-
 
|-
−
| Whole site.
+
| All Castor (all SRMs)
 
| SCHEDULED
 
| SCHEDULED
−
| WARNING
+
| OUTAGE
−
| 12/03/2015 07:45
+
| 08/04/2015 10:00
−
| 12/03/2015 08:30
+
| 08/04/2015 14:00
−
| 45 minutes
+
| 4 hours
−
| Warning following installation of a replacement network router.
+
|Upgrade of Castor storage to version 2.1.14-15
 
|-
 
|-
−
| Whole site.
+
| Whole site
 
| SCHEDULED
 
| SCHEDULED
 
| WARNING
 
| WARNING
−
| 11/03/2015 10:30
+
| 19/03/2015 07:45
−
| 11/03/2015 11:00
+
| 19/03/2015 08:30
−
| 30 minutes
+
| 45 minutes
−
| Warning on site access during firmware updates on pair of network switches.  
+
|Warning during tests of Tier1s network router. Possibility of two short (few minute) breaks in connectivity to the RAL Tier1.
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 106: Line 105:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
−
* Investigate problems on the primary Tier1 router. Discussions with the vendor are ongoing. (Swap with replacement unit planned for tomorrow morning, 12th March).
+
* Investigate problems on the primary Tier1 router. Further tests scheduled for tomorrow morning, 19th March.
 
* Update Castor to 2.1-14-15 (Proposed date - Wednesday 8th April).
 
* Update Castor to 2.1-14-15 (Proposed date - Wednesday 8th April).
 
'''Listing by category:'''
 
'''Listing by category:'''
Line 115: Line 114:
 
** Update Castor to 2.1.14-15 (Proposed date - Wednesday 8th April).
 
** Update Castor to 2.1.14-15 (Proposed date - Wednesday 8th April).
 
** Update SRMs to new version (includes updating to SL6).
 
** Update SRMs to new version (includes updating to SL6).
−
** Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
+
** Fix discrepancies found in some of the Castor database tables and columns. (The issue has no operational impact.)
 
* Networking:
 
* Networking:
 
** Resolve problems with primary Tier1 Router
 
** Resolve problems with primary Tier1 Router
Line 143: Line 142:
 
! Reason
 
! Reason
 
|-
 
|-
−
| Whole Site
+
| Whole site
 +
| SCHEDULED
 +
| WARNING
 +
| 12/03/2015 07:45
 +
| 12/03/2015 08:30
 +
| 45 minutes
 +
|Warning following installation of a replacement network router.
 +
|-
 +
| Whole site
 
| SCHEDULED
 
| SCHEDULED
 
| WARNING
 
| WARNING
Line 150: Line 157:
 
| 30 minutes
 
| 30 minutes
 
| Warning on site access during firmware updates on pair of network switches.  
 
| Warning on site access during firmware updates on pair of network switches.  
−
|-
 
−
| lcgfts3.gridpp.rl.ac.uk,
 
−
| SCHEDULED
 
−
| WARNING
 
−
| 05/03/2015 10:00
 
−
| 05/03/2015 12:00
 
−
| 2 hours
 
−
| Warning on FTS3 service during upgrade to version v3.2.32
 
−
|-
 
−
| Whole Site
 
−
| UNSCHEDULED
 
−
| WARNING
 
−
| 05/03/2015 07:45
 
−
| 05/03/2015 08:30
 
−
| 45 minutes
 
−
| Warning following installation of a replacement network router.
 
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 181: Line 172:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 +
|-
 +
| 112350
 +
| Green
 +
| Less Urgent
 +
| In Progress
 +
| 2015-03-15
 +
| 2015-03-17
 +
| SNO+
 +
| Download issue
 
|-  
 
|-  
 
| 112166
 
| 112166
Line 187: Line 187:
 
| In Progress
 
| In Progress
 
| 2015-03-06
 
| 2015-03-06
−
| 2015-03-11
+
| 2015-03-17
 
| Atlas
 
| Atlas
 
| RAL-LCG2 DATATAPE: destination transfer errors like "This SURL does not exist in the original request"
 
| RAL-LCG2 DATATAPE: destination transfer errors like "This SURL does not exist in the original request"
Line 196: Line 196:
 
| In Progress
 
| In Progress
 
| 2015-02-10
 
| 2015-02-10
−
| 2015-02-27
+
| 2015-03-11
 
| Atlas
 
| Atlas
 
| gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
 
| gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
Line 205: Line 205:
 
| In Progress
 
| In Progress
 
| 2014-11-03
 
| 2014-11-03
−
| 2015-03-10
+
| 2015-03-11
 
| SNO+
 
| SNO+
 
| gfal-copy failing for files at RAL
 
| gfal-copy failing for files at RAL
Line 211: Line 211:
 
| 108944
 
| 108944
 
| Red
 
| Red
−
| Urgent
+
| Less Urgent
−
| Waiting Reply
+
| In Progress
 
| 2014-10-01
 
| 2014-10-01
 
| 2015-03-03
 
| 2015-03-03
Line 238: Line 238:
 
| 11/03/15 || 100 || 100 || 100 || 100 || 100 || 98 || 100 ||
 
| 11/03/15 || 100 || 100 || 100 || 100 || 100 || 98 || 100 ||
 
|-
 
|-
−
| 12/03/15 || 100 || 100 || 100 || 95 || 100 || 100 || 100 || Problem with CE tests - possibly triggered by network intervention.
+
| 12/03/15 || 100 || 100 || 100 || style="background-color: lightgrey;" | 95 || 100 || 100 || 100 || Problem with CE tests - possibly triggered by network intervention.
 
|-
 
|-
−
| 13/03/15 || 100 || 100 || 96 || 96 || 96 || 100 || 100 || Remote monioring problem.
+
| 13/03/15 || 100 || 100 || style="background-color: lightgrey;" | 96 || style="background-color: lightgrey;" | 96 || style="background-color: lightgrey;" | 96 || 100 || 100 || Remote monitoring problem.
 
|-
 
|-
 
| 14/03/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
| 14/03/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||

Latest revision as of 14:05, 18 March 2015

RAL Tier1 Operations Report for 18th March 2015

Review of Issues during the week 11th to 18th March 2015.
  • The replacement of the the primary Tier1 router did not fix the problem whereby the router fails after a few minutes in service.
Resolved Disk Server Issues
  • None
Current operational status and issues
  • We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • Five additional disk servers have been deployed into Castor storage for ALICE_Disk. This roughly doubles the disk space available to ALICE which is now approximately 390TB.
  • On the CEs the upper memory limit was reduced from 5x requested memory to 3x.
  • ATLAS are now putting all UK transfers through our test FTS3.
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
All Castor (all SRMs) SCHEDULED OUTAGE 08/04/2015 10:00 08/04/2015 14:00 4 hours Upgrade of Castor storage to version 2.1.14-15
Whole site SCHEDULED WARNING 19/03/2015 07:45 19/03/2015 08:30 45 minutes Warning during tests of Tier1s network router. Possibility of two short (few minute) breaks in connectivity to the RAL Tier1.
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Investigate problems on the primary Tier1 router. Further tests scheduled for tomorrow morning, 19th March.
  • Update Castor to 2.1-14-15 (Proposed date - Wednesday 8th April).

Listing by category:

  • Databases:
    • Switch LFC/3D to new Database Infrastructure.
    • Update to Oracle 11.2.0.4
  • Castor:
    • Update Castor to 2.1.14-15 (Proposed date - Wednesday 8th April).
    • Update SRMs to new version (includes updating to SL6).
    • Fix discrepancies found in some of the Castor database tables and columns. (The issue has no operational impact.)
  • Networking:
    • Resolve problems with primary Tier1 Router
    • Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
    • Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
    • Make routing changes to allow the removal of the UKLight Router.
  • Fabric
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report.
Service Scheduled? Outage/At Risk Start End Duration Reason
Whole site SCHEDULED WARNING 12/03/2015 07:45 12/03/2015 08:30 45 minutes Warning following installation of a replacement network router.
Whole site SCHEDULED WARNING 11/03/2015 10:30 11/03/2015 11:00 30 minutes Warning on site access during firmware updates on pair of network switches.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
112350 Green Less Urgent In Progress 2015-03-15 2015-03-17 SNO+ Download issue
112166 Green Urgent In Progress 2015-03-06 2015-03-17 Atlas RAL-LCG2 DATATAPE: destination transfer errors like "This SURL does not exist in the original request"
111699 Green Less Urgent In Progress 2015-02-10 2015-03-11 Atlas gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
109694 Red Urgent In Progress 2014-11-03 2015-03-11 SNO+ gfal-copy failing for files at RAL
108944 Red Less Urgent In Progress 2014-10-01 2015-03-03 CMS AAA access test failing at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
11/03/15 100 100 100 100 100 98 100
12/03/15 100 100 100 95 100 100 100 Problem with CE tests - possibly triggered by network intervention.
13/03/15 100 100 96 96 96 100 100 Remote monitoring problem.
14/03/15 100 100 100 100 100 100 100
15/03/15 100 100 100 100 100 100 99
16/03/15 100 100 100 100 100 100 100
17/03/15 100 100 100 100 100 100 99
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2015-03-18&oldid=7988"