Difference between revisions of "Tier1 Operations Report 2019-09-18"

From GridPP Wiki
Jump to: navigation, search
(Created page with "==RAL Tier1 Operations Report for 18th September 2019== __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Sta...")
 
()
 
(4 intermediate revisions by one user not shown)
Line 10: Line 10:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 11th September 2019 to the 18th September 2019.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 11th September 2019 to the 18th September 2019.
 
|}
 
|}
* TBA
+
* NTR
  
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
Line 56: Line 56:
 
! Reason
 
! Reason
 
|-
 
|-
|All
 
|27719
 
|Yes
 
|At Risk
 
|11-09-19 0530
 
|11-09-19 0700
 
|90mins
 
| Network routing hardware link change
 
 
|-
 
|-
 
+
|-
 +
|-
 +
|-
 +
|-
 +
|-
 +
|-
 +
|-
 +
|-
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 126: Line 125:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open GGUS Tickets  
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open GGUS Tickets  
 
|}
 
|}
 
 
 
{| border=1 align=center
 
{| border=1 align=center
 
|- bgcolor="#7c8aaf"
 
|- bgcolor="#7c8aaf"
Line 140: Line 137:
 
! Subject
 
! Subject
 
! Scope
 
! Scope
 +
|-
 +
| 143231
 +
| USER
 +
| other
 +
| RAL-LCG2
 +
| urgent
 +
| EGI CVMFS Service
 +
| in progress
 +
| 2019-09-18 09:28:00
 +
| CVMFS repo dirac.egi.eu updates are not propagated
 +
| EGI
 +
|-
 +
| 143225
 +
| USER
 +
| cms
 +
| RAL-LCG2
 +
| very urgent
 +
| NGI_UK
 +
| in progress
 +
| 2019-09-17 08:25:00
 +
| some of RAL FTS servers are not running?
 +
| WLCG
 +
|-
 +
| 143218
 +
| TEAM
 +
| lhcb
 +
| RAL-LCG2
 +
| urgent
 +
| NGI_UK
 +
| in progress
 +
| 2019-09-16 06:45:00
 +
| FTS3 transfers problem to GRIDKA for transfers executing at RAL FTS3 server
 +
| WLCG
 
|-
 
|-
 
| 142835
 
| 142835
Line 147: Line 177:
 
| less urgent
 
| less urgent
 
| NGI_UK
 
| NGI_UK
| waiting for reply
+
| in progress
| 2019-09-09 14:16:00
+
| 2019-09-16 14:41:00
 
| Connection Issues
 
| Connection Issues
 
| EGI
 
| EGI
Line 158: Line 188:
 
| very urgent
 
| very urgent
 
| NGI_UK
 
| NGI_UK
| in progress
+
| waiting for reply
| 2019-09-02 17:22:00
+
| 2019-09-12 12:09:00
 
| Transfer failing to RAL_Disk
 
| Transfer failing to RAL_Disk
 
| WLCG
 
| WLCG
Line 170: Line 200:
 
| NGI_UK
 
| NGI_UK
 
| in progress
 
| in progress
| 2019-09-03 12:41:00
+
| 2019-09-17 10:03:00
 
| Proble accessing some LHCb files at RAL
 
| Proble accessing some LHCb files at RAL
 
| WLCG
 
| WLCG
Line 180: Line 210:
 
| less urgent
 
| less urgent
 
| NGI_UK
 
| NGI_UK
| on hold
+
| in progress
| 2019-08-22 10:04:00
+
| 2019-09-11 10:37:00
 
| packet loss outbound from RAL-LCG2 over IPv6
 
| packet loss outbound from RAL-LCG2 over IPv6
 
| EGI
 
| EGI
 
|}
 
|}
 
 
 
 
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->
Line 201: Line 227:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | GGUS Tickets Closed Last week
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | GGUS Tickets Closed Last week
 
|}
 
|}
 
 
 
{| border=1 align=center
 
{| border=1 align=center
 
|- bgcolor="#7c8aaf"
 
|- bgcolor="#7c8aaf"
Line 216: Line 240:
 
! Scope
 
! Scope
 
|-
 
|-
| 142981
+
| 143198
 
| USER
 
| USER
| mice
+
| cms
 
| RAL-LCG2
 
| RAL-LCG2
| less urgent
+
| urgent
 
| NGI_UK
 
| NGI_UK
 
| solved
 
| solved
| 2019-09-04 13:02:00
+
| 2019-09-13 13:54:00
| mice; LFC to DFC transition
+
| issues with RAL FTS?
| EGI
+
|-
+
| 142955
+
| USER
+
| ops
+
| RAL-LCG2
+
| less urgent
+
| NGI_UK
+
| verified
+
| 2019-09-05 13:15:00
+
| [Rod Dashboard] Issues detected at RAL-LCG2
+
| EGI
+
|-
+
| 142782
+
| TEAM
+
| lhcb
+
| RAL-LCG2
+
| very urgent
+
| NGI_UK
+
| verified
+
| 2019-09-04 12:32:00
+
| FTS3 transfers Failed to RAL-RDST at RAL-LCG2
+
 
| WLCG
 
| WLCG
 
|-
 
|-
| 142751
+
| 142815
 
| USER
 
| USER
| snoplus.snolab.ca
+
| cms
 
| RAL-LCG2
 
| RAL-LCG2
| top priority
+
| urgent
 
| NGI_UK
 
| NGI_UK
 
| closed
 
| closed
| 2019-09-04 23:59:00
+
| 2019-09-12 23:59:00
| Data transfer failure and proxy issue
+
| PhEDEx deletions pending since 10+ days at T1_UK_RAL_Disk
| EGI
+
| WLCG
 
|}
 
|}
 
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->
Line 283: Line 284:
 
! Comments
 
! Comments
 
|-
 
|-
| 2019-09-04
+
| 2019-09-11
 
| 100
 
| 100
 +
| 99
 
| 100
 
| 100
 
| 100
 
| 100
| 91
 
 
|  
 
|  
 
|-
 
|-
| 2019-09-05
+
| 2019-09-12
 +
| 100
 +
| 100
 +
| 100
 +
| 100
 +
|
 +
|-
 +
| 2019-09-13
 
| 100
 
| 100
 
| 99
 
| 99
Line 297: Line 305:
 
|  
 
|  
 
|-
 
|-
| 2019-09-06
+
| 2019-09-14
 
| 100
 
| 100
 
| 100
 
| 100
Line 304: Line 312:
 
|  
 
|  
 
|-
 
|-
| 2019-09-07
+
| 2019-09-15
 +
| 100
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
| 91
 
| 92
 
 
|  
 
|  
 
|-
 
|-
| 2019-09-08
+
| 2019-09-16
 +
| 100
 +
| 99
 
| 100
 
| 100
 
| 100
 
| 100
| 0
 
| 0
 
 
|  
 
|  
 
|-
 
|-
| 2019-09-09
+
| 2019-09-17
 +
| 100
 +
| 99
 
| 100
 
| 100
 
| 100
 
| 100
| 63
 
| 64
 
 
|  
 
|  
 
|-
 
|-
| 2019-09-10
+
| 2019-09-18
| 100
+
 
| 100
 
| 100
 +
| 97
 
| 100
 
| 100
 
| 100
 
| 100

Latest revision as of 11:53, 18 September 2019

RAL Tier1 Operations Report for 18th September 2019

Review of Issues during the week 11th September 2019 to the 18th September 2019.
  • NTR


Current operational status and issues
Notable Changes made since the last meeting.
  • NTR
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.


Listing by category:

  • DNS servers will be rolled out within the Tier1 network.
Open GGUS Tickets
Ticket-ID Type VO Site Priority Responsible Unit Status Last Update Subject Scope
143231 USER other RAL-LCG2 urgent EGI CVMFS Service in progress 2019-09-18 09:28:00 CVMFS repo dirac.egi.eu updates are not propagated EGI
143225 USER cms RAL-LCG2 very urgent NGI_UK in progress 2019-09-17 08:25:00 some of RAL FTS servers are not running? WLCG
143218 TEAM lhcb RAL-LCG2 urgent NGI_UK in progress 2019-09-16 06:45:00 FTS3 transfers problem to GRIDKA for transfers executing at RAL FTS3 server WLCG
142835 USER snoplus.snolab.ca RAL-LCG2 less urgent NGI_UK in progress 2019-09-16 14:41:00 Connection Issues EGI
142689 USER cms RAL-LCG2 very urgent NGI_UK waiting for reply 2019-09-12 12:09:00 Transfer failing to RAL_Disk WLCG
142350 TEAM lhcb RAL-LCG2 top priority NGI_UK in progress 2019-09-17 10:03:00 Proble accessing some LHCb files at RAL WLCG
140447 USER dteam RAL-LCG2 less urgent NGI_UK in progress 2019-09-11 10:37:00 packet loss outbound from RAL-LCG2 over IPv6 EGI
GGUS Tickets Closed Last week
Ticket-ID Type VO Site Priority Responsible Unit Status Last Update Subject Scope
143198 USER cms RAL-LCG2 urgent NGI_UK solved 2019-09-13 13:54:00 issues with RAL FTS? WLCG
142815 USER cms RAL-LCG2 urgent NGI_UK closed 2019-09-12 23:59:00 PhEDEx deletions pending since 10+ days at T1_UK_RAL_Disk WLCG

Availability Report

Day Atlas CMS LHCB Alice Comments
2019-09-11 100 99 100 100
2019-09-12 100 100 100 100
2019-09-13 100 99 100 100
2019-09-14 100 100 100 100
2019-09-15 100 100 100 100
2019-09-16 100 99 100 100
2019-09-17 100 99 100 100
2019-09-18 100 97 100 100
Hammercloud Test Report
Target Availability for each site is 97.0%
Day Atlas HC CMS HC Comment
2019-09-04 96 99
2019-09-05 100 99
2019-09-06 100 98
2019-09-07 100 99
2019-09-08 100 100
2019-09-09 93 100
2019-09-10 100 100

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.