Difference between revisions of "Tier1 Operations Report 2019-09-04"

From GridPP Wiki
Jump to: navigation, search
()
()
Line 225: Line 225:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | GGUS Tickets Closed Last week
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | GGUS Tickets Closed Last week
 
|}
 
|}
 +
 +
 
{| border=1 align=center
 
{| border=1 align=center
 
|- bgcolor="#7c8aaf"
 
|- bgcolor="#7c8aaf"
Line 238: Line 240:
 
! Scope
 
! Scope
 
|-
 
|-
| 142751
+
| 142815
 
| USER
 
| USER
| snoplus.snolab.ca
+
| cms
| RAL-LCG2
+
| top priority
+
| NGI_UK
+
| solved
+
| 2019-08-21 08:39:00
+
| Data transfer failure and proxy issue
+
| EGI
+
|-
+
| 142694
+
| TEAM
+
| atlas
+
 
| RAL-LCG2
 
| RAL-LCG2
 
| urgent
 
| urgent
 
| NGI_UK
 
| NGI_UK
 
| solved
 
| solved
| 2019-08-14 09:10:00
+
| 2019-08-29 14:28:00
| RAL-LCG2 transfer errors at source
+
| PhEDEx deletions pending since 10+ days at T1_UK_RAL_Disk
 
| WLCG
 
| WLCG
 
|-
 
|-
| 142665
+
| 142782
| USER
+
| TEAM
| cms
+
| lhcb
 
| RAL-LCG2
 
| RAL-LCG2
| urgent
+
| very urgent
 
| NGI_UK
 
| NGI_UK
 
| solved
 
| solved
| 2019-08-14 09:32:00
+
| 2019-08-30 09:34:00
| Failing to transfer few files to RAL_Disk from CERN
+
| FTS3 transfers Failed to RAL-RDST at RAL-LCG2
| WLCG
+
|-
+
| 142520
+
| USER
+
| cms
+
| RAL-LCG2
+
| urgent
+
| NGI_UK
+
| closed
+
| 2019-08-14 23:59:00
+
| T1_UK_RAL is failing SAM tests
+
 
| WLCG
 
| WLCG
 
|-
 
|-
| 142337
+
| 142710
 
| TEAM
 
| TEAM
 
| lhcb
 
| lhcb
Line 289: Line 269:
 
| NGI_UK
 
| NGI_UK
 
| verified
 
| verified
| 2019-08-14 15:10:00
+
| 2019-08-30 09:51:00
| Pilots Failed at RAL-LCG2
+
| Staging problems
 
| WLCG
 
| WLCG
 
|-
 
|-
| 142203
+
| 142694
 
| TEAM
 
| TEAM
 
| atlas
 
| atlas
Line 300: Line 280:
 
| NGI_UK
 
| NGI_UK
 
| closed
 
| closed
| 2019-08-14 23:59:00
+
| 2019-08-28 23:59:00
| RAL-LCG2_MCORE jobs failing
+
| RAL-LCG2 transfer errors at source
 +
| WLCG
 +
|-
 +
| 142665
 +
| USER
 +
| cms
 +
| RAL-LCG2
 +
| urgent
 +
| NGI_UK
 +
| closed
 +
| 2019-08-28 23:59:00
 +
| Failing to transfer few files to RAL_Disk from CERN
 
| WLCG
 
| WLCG
 
|-
 
|-
Line 310: Line 301:
 
| less urgent
 
| less urgent
 
| NGI_UK
 
| NGI_UK
| solved
+
| closed
| 2019-08-14 19:09:00
+
| 2019-08-28 23:59:00
 
| mice LFC to DFC transition
 
| mice LFC to DFC transition
 
| EGI
 
| EGI

Revision as of 10:06, 4 September 2019

RAL Tier1 Operations Report for 04th September 2019

Review of Issues during the week 25th July2019 to the 31st July 2019.
  • ATLAS and LHCb keeping robot busy with recall campaigns.
  • Power station demolition caused cloud resources to be offline, grid hardware ok.
  • lsst jobs runnning after fixing unexpected change of voms certificate.


Current operational status and issues
Notable Changes made since the last meeting.
  • NTR
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.


Listing by category:

  • DNS servers will be rolled out within the Tier1 network.
Open GGUS Tickets


Ticket-ID Type VO Site Priority Responsible Unit Status Last Update Subject Scope
142981 USER mice RAL-LCG2 less urgent NGI_UK in progress 2019-09-03 13:00:00 mice; LFC to DFC transition EGI
142955 USER ops RAL-LCG2 less urgent NGI_UK in progress 2019-09-02 10:26:00 [Rod Dashboard] Issues detected at RAL-LCG2 EGI
142835 USER snoplus.snolab.ca RAL-LCG2 less urgent NGI_UK waiting for reply 2019-08-30 09:25:00 Connection Issues EGI
142689 USER cms RAL-LCG2 very urgent NGI_UK in progress 2019-09-02 17:22:00 Transfer failing to RAL_Disk WLCG
142350 TEAM lhcb RAL-LCG2 top priority NGI_UK in progress 2019-09-03 12:41:00 Proble accessing some LHCb files at RAL WLCG
140447 USER dteam RAL-LCG2 less urgent NGI_UK on hold 2019-08-22 10:04:00 packet loss outbound from RAL-LCG2 over IPv6 EGI




GGUS Tickets Closed Last week


Ticket-ID Type VO Site Priority Responsible Unit Status Last Update Subject Scope
142815 USER cms RAL-LCG2 urgent NGI_UK solved 2019-08-29 14:28:00 PhEDEx deletions pending since 10+ days at T1_UK_RAL_Disk WLCG
142782 TEAM lhcb RAL-LCG2 very urgent NGI_UK solved 2019-08-30 09:34:00 FTS3 transfers Failed to RAL-RDST at RAL-LCG2 WLCG
142710 TEAM lhcb RAL-LCG2 very urgent NGI_UK verified 2019-08-30 09:51:00 Staging problems WLCG
142694 TEAM atlas RAL-LCG2 urgent NGI_UK closed 2019-08-28 23:59:00 RAL-LCG2 transfer errors at source WLCG
142665 USER cms RAL-LCG2 urgent NGI_UK closed 2019-08-28 23:59:00 Failing to transfer few files to RAL_Disk from CERN WLCG
140220 USER mice RAL-LCG2 less urgent NGI_UK closed 2019-08-28 23:59:00 mice LFC to DFC transition EGI


Availability Report

Day Atlas CMS LHCB Alice Comments
2019-08-14 100 99 100 100
2019-08-15 100 100 67 68
2019-08-16 100 100 48 48
2019-08-17 100 100 100 100
2019-08-18 100 100 100 100
2019-08-19 100 100 100 100
2019-08-20 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0%
Day Atlas HC CMS HC Comment
2019-08-14 100 99
2019-08-15 100 99
2019-08-16 100 98
2019-08-17 100 99
2019-08-18 100 98
2019-08-19 100 100
2019-08-20 0 100

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.