Difference between revisions of "Tier1 Operations Report 2019-01-07"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(20 intermediate revisions by one user not shown)
Line 10: Line 10:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 2nd January 2019 to the 7th January 2019.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 2nd January 2019 to the 7th January 2019.
 
|}
 
|}
−
*  
+
* Very smooth running over Christmas period.
 +
* 28th December, File System problem with one of the Squids, was fixed on 2nd Jan.
 +
* Problem with one of the ARC CEs, this seems to be intermittently impacting VOs.  Currently under investigation.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 40: Line 42:
 
! Comments
 
! Comments
 
|-
 
|-
−
| -
+
| gdss761
−
| -
+
| lhcb
−
| -
+
| lhcbDst
−
| -
+
| d1t0
 
| -
 
| -
 
|}
 
|}
Line 63: Line 65:
 
! Comments
 
! Comments
 
|-
 
|-
−
| fdsdss51
+
| gdss804
−
| Facilities
+
| lhcb
−
| diamondRecall
+
| lhcbDst
−
| d0t0
+
| d1t0
 
| -
 
| -
 
|}
 
|}
Line 189: Line 191:
 
! Scope
 
! Scope
 
|-
 
|-
−
| style="background-color: lightgreen;" | 138891
+
| style="background-color: lightgreen;" | 139108
 
| ops
 
| ops
 
| in progress
 
| in progress
 +
| less urgent
 +
| 09/01/2019
 +
| 09/01/2019
 +
| Operations
 +
| [Rod Dashboard] Issue detected : org.nordugrid.ARC-CE-ARIS@arc-ce04.gridpp.rl.ac.uk
 +
| EGI
 +
|-
 +
| style="background-color: lightgreen;" | 139106
 +
| lhcb
 +
| in progress
 +
| very urgent
 +
| 09/01/2019
 +
| 09/01/2019
 +
| File Transfer
 +
| FTS3 transfer problem at RAL-LCG2
 +
| WLCG
 +
|-
 +
| style="background-color: lightgreen;" | 139081
 +
| lhcb
 +
| in progress
 +
| top priority
 +
| 07/01/2019
 +
| 07/01/2019
 +
| Local Batch System
 +
| Aborted pilots for LHCb
 +
| WLCG
 +
|-
 +
| style="background-color: lightgreen;" | 138891
 +
| ops
 +
| on hold
 
| less urgent
 
| less urgent
 
| 17/12/2018
 
| 17/12/2018
−
| 17/12/2018
+
| 07/01/2019
 
| Operations
 
| Operations
 
| [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability
 
| [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability
Line 204: Line 236:
 
| urgent
 
| urgent
 
| 13/12/2018
 
| 13/12/2018
−
| 13/12/2018
+
| 08/01/2019
 
| Storage Systems
 
| Storage Systems
 
| Tape reporting is not being updated
 
| Tape reporting is not being updated
Line 244: Line 276:
 
| urgent
 
| urgent
 
| 01/11/2018
 
| 01/11/2018
−
| 30/11/2018
+
| 08/01/2019
 
| Other
 
| Other
 
| singularity jobs failing at RAL
 
| singularity jobs failing at RAL
Line 254: Line 286:
 
| less urgent
 
| less urgent
 
| 23/10/2018
 
| 23/10/2018
−
| 19/12/2018
+
| 07/01/2019
 
| Accounting
 
| Accounting
 
| enmr.eu accounting at RAL
 
| enmr.eu accounting at RAL
Line 283: Line 315:
 
! Scope
 
! Scope
 
|-
 
|-
−
| 138948
+
| 139034
 
| cms
 
| cms
 
| solved
 
| solved
 
| urgent
 
| urgent
−
| 19/12/2018
+
| 31/12/2018
−
| 20/12/2018
+
| 02/01/2019
 
| CMS_Facilities
 
| CMS_Facilities
−
| T1_UK_RAL SRM SAM tests failing (disk pool is full)
+
| T1_UK_RAL squid05 not responding
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 138825
+
| 139031
−
| cms
+
| lhcb
−
| solved
+
| verified
 
| urgent
 
| urgent
−
| 12/12/2018
+
| 31/12/2018
−
| 19/12/2018
+
| 02/01/2019
−
| CMS_Data Transfers
+
| File Transfer
−
| File staging at T1_UK_RAL_Buffer
+
| FTS3 transfer problem from RAL
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 138815
+
| 139020
−
| cms
+
| atlas
−
| closed
+
| solved
−
| urgent
+
| less urgent
−
| 12/12/2018
+
| 29/12/2018
−
| 26/12/2018
+
| 02/01/2019
−
| CMS_Data Transfers
+
| Other
−
| Possible expired proxy at RAL
+
| RAL-LCG2 squid service degraded
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 138802
+
| 139015
−
| atlas
+
| lhcb
 
| verified
 
| verified
 
| very urgent
 
| very urgent
−
| 11/12/2018
 
−
| 20/12/2018
 
−
| Local Batch System
 
−
| Checksum failures when reading job input
 
−
| EGI
 
−
|-
 
−
| 138762
 
−
| cms
 
−
| closed
 
−
| urgent
 
−
| 10/12/2018
 
−
| 27/12/2018
 
−
| CMS_Data Transfers
 
−
| Transfers failing from FNAL to RAL_Disk
 
−
| WLCG
 
−
|-
 
−
| 138760
 
−
| cms
 
−
| closed
 
−
| urgent
 
−
| 10/12/2018
 
 
| 28/12/2018
 
| 28/12/2018
−
| CMS_Data Transfers
+
| 02/01/2019
−
| Transfers failing from RAL to CCIN2P3_Disk
+
| Other
 +
| arc-ce04.gridpp.rl.ac.uk unavailable at RAL-LCG2
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 138758
+
| 138948
−
| none
+
−
| verified
+
−
| less urgent
+
−
| 10/12/2018
+
−
| 20/12/2018
+
−
| File Transfer
+
−
| RAL-LCG2-ECHO: No such file or directory
+
−
| EGI
+
−
|-
+
−
| 138736
+
 
| cms
 
| cms
 
| closed
 
| closed
 
| urgent
 
| urgent
−
| 07/12/2018
+
| 19/12/2018
−
| 25/12/2018
+
| 03/01/2019
 
| CMS_Facilities
 
| CMS_Facilities
−
| T1_UK_RAL intermittent SRM VOGet/VOPut failures
+
| T1_UK_RAL SRM SAM tests failing (disk pool is full)
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 138715
+
| 138825
−
| atlas
+
−
| closed
+
−
| urgent
+
−
| 05/12/2018
+
−
| 21/12/2018
+
−
| File Transfer
+
−
| RAL-LCG2-ECHO: Transfer errors as destintion
+
−
| WLCG
+
−
|-
+
−
| 138663
+
 
| cms
 
| cms
 
| closed
 
| closed
 
| urgent
 
| urgent
−
| 04/12/2018
+
| 12/12/2018
−
| 18/12/2018
+
| 02/01/2019
 
| CMS_Data Transfers
 
| CMS_Data Transfers
−
| Transfers failing from Swierk to RAL
+
| File staging at T1_UK_RAL_Buffer
−
| WLCG
+
−
|-
+
−
| 138613
+
−
| cms
+
−
| closed
+
−
| urgent
+
−
| 29/11/2018
+
−
| 19/12/2018
+
−
| CMS_Data Transfers
+
−
| RAL Staging request
+
−
| WLCG
+
−
|-
+
−
| 138584
+
−
| cms
+
−
| closed
+
−
| urgent
+
−
| 28/11/2018
+
−
| 20/12/2018
+
−
| CMS_AAA WAN Access
+
−
| T1_UK_RAL xrootd reads timing out
+
−
| WLCG
+
−
|-
+
−
| 138493
+
−
| cms
+
−
| closed
+
−
| very urgent
+
−
| 26/11/2018
+
−
| 24/12/2018
+
−
| CMS_Data Transfers
+
−
| All Transfers failing from RAL_Disk
+
−
| WLCG
+
−
|-
+
−
| 137822
+
−
| lhcb
+
−
| closed
+
−
| top priority
+
−
| 18/10/2018
+
−
| 18/12/2018
+
−
| File Transfer
+
−
| FTS server seems in bad state.
+
 
| WLCG
 
| WLCG
 
|}
 
|}
Line 452: Line 404:
 
! Comments
 
! Comments
 
|-
 
|-
−
| 2018-12-18
+
| 2019-01-01
 
| 100
 
| 100
 
| 100
 
| 100
Line 458: Line 410:
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: cyan;" | -1
 
 
|  
 
|  
 
|-
 
|-
−
| 2018-12-19
+
| 2019-01-02
−
| 100
+
−
| 100
+
−
| style="background-color: red;" | 86
+
−
| 100
+
−
| 100
+
−
| style="background-color: cyan;" | 0
+
−
|
+
−
|-
+
−
| 2018-12-20
+
 
| 100
 
| 100
 
| 100
 
| 100
Line 476: Line 418:
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: cyan;" | -1
+
| style="background-color: lightblue;" | -1
 
|  
 
|  
 
|-
 
|-
−
| 2018-12-21
+
| 2019-01-03
 
| 100
 
| 100
 
| 100
 
| 100
Line 485: Line 427:
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: cyan;" | -1
+
| style="background-color: lightblue;" | -1
 
|  
 
|  
 
|-
 
|-
−
| 2018-12-22
+
| 2019-01-04
 
| 100
 
| 100
 
| 100
 
| 100
Line 494: Line 436:
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: cyan;" | -1
+
| style="background-color: lightblue;" | -1
 
|  
 
|  
 
|-
 
|-
−
| 2018-12-23
+
| 2019-01-05
 
| 100
 
| 100
 
| 100
 
| 100
Line 503: Line 445:
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: cyan;" | -1
+
| style="background-color: lightblue;" | -1
 
|  
 
|  
 
|-
 
|-
−
| 2018-12-24
+
| 2019-01-06
 
| 100
 
| 100
 
| 100
 
| 100
Line 512: Line 454:
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: cyan;" | -1
+
| style="background-color: lightblue;" | -1
 
|  
 
|  
 
|-
 
|-
−
| 2018-12-25
+
| 2019-01-07
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 +
| 98
 
| 100
 
| 100
−
| 100
+
| style="background-color: lightblue;" | -1
−
| style="background-color: cyan;" | -1
+
 
|  
 
|  
 
|-
 
|-
−
| 2018-12-26
+
| 2019-01-08
 
| 100
 
| 100
 
| 100
 
| 100
Line 530: Line 472:
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: cyan;" | -1
+
| style="background-color: lightblue;" | -1
−
|
+
−
|-
+
−
| 2018-12-27
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| style="background-color: cyan;" | -1
+
−
|
+
−
|-
+
−
| 2018-12-28
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| style="background-color: cyan;" | -1
+
−
|
+
−
|-
+
−
| 2018-12-29
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| style="background-color: cyan;" | -1
+
−
|
+
−
|-
+
−
| 2018-12-30
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| style="background-color: cyan;" | -1
+
−
|
+
−
|-
+
−
| 2018-12-31
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
|
+
−
|-
+
−
| 2019-01-01
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
|
+
−
|-
+
−
| 2019-01-02
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| style="background-color: cyan;" | -1
+
 
|  
 
|  
 +
 
|}
 
|}
  
Line 611: Line 493:
 
! Day !! Atlas HC !! CMS HC !! Comment
 
! Day !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 2018-12-17 || 100 || 100|| 
+
| 2019-01-01 || 100 || 100 ||  
−
|-
+
−
| 2018-12-18 || 100 || 100 ||
+
−
|-
+
−
| 2018-12-19 || 100 || 100 ||
+
−
|-
+
−
| 2018-12-20 || 100 || 100 ||
+
−
|-
+
−
| 2018-12-21 || 100 || 100 ||
+
−
|-
+
−
| 2018-12-22 || 100 || 100 ||
+
−
|-
+
−
| 2018-12-23 || 100 || 100 ||
+
−
|-
+
−
| 2018-12-24 || 100 || style="background-color: orange;" | 93  ||
+
−
|-
+
−
| 2018-12-25 || 100 || 99 ||  
+
 
|-
 
|-
−
| 2018-12-26 || 100 || N/A ||  
+
| 2019-01-02 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-12-27 || 100 || N/A ||  
+
| 2019-01-03 || 99 || 99 ||  
 
|-
 
|-
−
| 2018-12-28 || 100 || N/A ||  
+
| 2019-01-04 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-12-29 || 100 || N/A ||  
+
| 2019-01-05 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-12-30 || 100 || N/A ||  
+
| 2019-01-06 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-12-31 || 100 || N/A ||  
+
| 2019-01-07 || 99 || 100 ||  
 
|-
 
|-
−
| 2019-01-01 || 100 || N/A ||  
+
| 2019-01-08 || 99 || 100 ||  
 
|-
 
|-
 
|}  
 
|}  

Latest revision as of 10:15, 9 January 2019

RAL Tier1 Operations Report for 7th January 2019

Review of Issues during the week 2nd January 2019 to the 7th January 2019.
  • Very smooth running over Christmas period.
  • 28th December, File System problem with one of the Squids, was fixed on 2nd Jan.
  • Problem with one of the ARC CEs, this seems to be intermittently impacting VOs. Currently under investigation.
Current operational status and issues
  • NTR
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss761 lhcb lhcbDst d1t0 -
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss804 lhcb lhcbDst d1t0 -
Limits on concurrent batch system jobs.
  • ALICE - 1000
Notable Changes made since the last meeting.
  • NTR
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - -
  • No ongoing downtime
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
139108 ops in progress less urgent 09/01/2019 09/01/2019 Operations [Rod Dashboard] Issue detected : org.nordugrid.ARC-CE-ARIS@arc-ce04.gridpp.rl.ac.uk EGI
139106 lhcb in progress very urgent 09/01/2019 09/01/2019 File Transfer FTS3 transfer problem at RAL-LCG2 WLCG
139081 lhcb in progress top priority 07/01/2019 07/01/2019 Local Batch System Aborted pilots for LHCb WLCG
138891 ops on hold less urgent 17/12/2018 07/01/2019 Operations [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability EGI
138833 none in progress urgent 13/12/2018 08/01/2019 Storage Systems Tape reporting is not being updated EGI
138665 mice in progress urgent 04/12/2018 12/12/2018 Middleware Problem accessing LFC at RAL EGI
138500 cms in progress urgent 26/11/2018 20/12/2018 CMS_Data Transfers Transfers failing from T2_PL_Swierk to RAL WLCG
138361 t2k.org in progress less urgent 19/11/2018 19/12/2018 Other RAL-LCG2: t2k.org LFC to DFC transition EGI
138033 atlas in progress urgent 01/11/2018 08/01/2019 Other singularity jobs failing at RAL EGI
137897 enmr.eu in progress less urgent 23/10/2018 07/01/2019 Accounting enmr.eu accounting at RAL EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
139034 cms solved urgent 31/12/2018 02/01/2019 CMS_Facilities T1_UK_RAL squid05 not responding WLCG
139031 lhcb verified urgent 31/12/2018 02/01/2019 File Transfer FTS3 transfer problem from RAL WLCG
139020 atlas solved less urgent 29/12/2018 02/01/2019 Other RAL-LCG2 squid service degraded WLCG
139015 lhcb verified very urgent 28/12/2018 02/01/2019 Other arc-ce04.gridpp.rl.ac.uk unavailable at RAL-LCG2 WLCG
138948 cms closed urgent 19/12/2018 03/01/2019 CMS_Facilities T1_UK_RAL SRM SAM tests failing (disk pool is full) WLCG
138825 cms closed urgent 12/12/2018 02/01/2019 CMS_Data Transfers File staging at T1_UK_RAL_Buffer WLCG

Availability Report

Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2019-01-01 100 100 100 100 100
2019-01-02 100 100 100 100 100 -1
2019-01-03 100 100 100 100 100 -1
2019-01-04 100 100 100 100 100 -1
2019-01-05 100 100 100 100 100 -1
2019-01-06 100 100 100 100 100 -1
2019-01-07 100 100 100 98 100 -1
2019-01-08 100 100 100 100 100 -1
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2019-01-01 100 100
2019-01-02 100 100
2019-01-03 99 99
2019-01-04 100 100
2019-01-05 100 100
2019-01-06 100 100
2019-01-07 99 100
2019-01-08 99 100

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2019-01-07&oldid=19220"