Difference between revisions of "Tier1 Operations Report 2018-09-24"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(10 intermediate revisions by one user not shown)
Line 11: Line 11:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 17th August to the 24th September 2018.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 17th August to the 24th September 2018.
 
|}
 
|}
−
* There has been improvements in ATLAS and their fair-share usage. This is ongoing work with the aim of attaining further gains.
+
* Tier-1 have spent the last week in the process of trying to resolve the on-going LFC issues.  
−
* There are currently a worrying number of GGUS tickets relating to the LFC. This is currently priority tasking for Tier-1 at this time.
+
* Our current understanding of the problem is that we have a corrupted LFC database. More correctly we believe a value (or values), within that database is causing an issue. We don't often see metadata showing a file-size of 18446744073709394047 bytes (yes, we think that's a yottabyte or two), and feel this may be causing some issues!
−
 
+
* We have restored back to a version of the database that works with gfal-copy (i.e. doesn't throw the error people were encountering) and are now working out what's missing and corrupted between the latest working and rolled back version of the database.
 +
* This issue was successfully resolve with the removal of the GOCDB unscheduled down-time at 12:00BST 24/9/18.
 +
* A number of associated LFC tickets are also being worked on in parallel.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 68: Line 70:
 
|-
 
|-
 
| gdss747  
 
| gdss747  
 +
| Atlas
 +
| atlasStripInput
 +
| d1t0
 +
| Currently in intervention.
 +
|-
 +
| gdss753
 
| Atlas  
 
| Atlas  
 
| atlasStripInput
 
| atlasStripInput
Line 118: Line 126:
 
! Reason
 
! Reason
 
|-
 
|-
 +
| LFC
 +
| 26064
 +
| No
 +
| Outage
 +
| 20-Sep-18 17:00:00
 +
| 24-Sep-18 11:00:02
 
| -
 
| -
−
| -
+
| investigating problems with the LFC - possibly in the back end database.
−
| -
+
−
| -
+
−
| -
+
−
| -
+
−
| -
+
−
| -
+
−
 
+
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 204: Line 211:
 
! Subject
 
! Subject
 
! Scope
 
! Scope
 +
|-
 +
| style="background-color: green;" | 137398
 +
| cms
 +
| in progress
 +
| urgent
 +
| 26/09/2018
 +
| 26/09/2018
 +
| CMS_Data Transfers
 +
| Transfers failing from SPRACE to RAL - No data available
 +
| WLCG
 +
|-
 +
| style="background-color: green;" | 137391
 +
| atlas
 +
| in progress
 +
| urgent
 +
| 25/09/2018
 +
| 26/09/2018
 +
| Network problem
 +
| UK RAL-LCG2 transfer errors with Communication error on send
 +
| WLCG
 
|-
 
|-
 
| style="background-color: green;" | 137254
 
| style="background-color: green;" | 137254
Line 210: Line 237:
 
| less urgent
 
| less urgent
 
| 18/09/2018
 
| 18/09/2018
−
| 24/09/2018
+
| 25/09/2018
 
| Operations
 
| Operations
 
| [Rod Dashboard] Issue detected : ch.cern.LFC-Write-ops@lfc.gridpp.rl.ac.uk
 
| [Rod Dashboard] Issue detected : ch.cern.LFC-Write-ops@lfc.gridpp.rl.ac.uk
Line 230: Line 257:
 
| urgent
 
| urgent
 
| 12/09/2018
 
| 12/09/2018
−
| 20/09/2018
+
| 25/09/2018
 
| Data Management - generic
 
| Data Management - generic
 
| LFC entry has file size 0, preventsw registering of additional replicas
 
| LFC entry has file size 0, preventsw registering of additional replicas
−
| EGI
 
−
|-
 
−
| style="background-color: green;" | 137151
 
−
| t2k.org
 
−
| in progress
 
−
| urgent
 
−
| 12/09/2018
 
−
| 20/09/2018
 
−
| File Transfer
 
−
| Cannot upload files to grid an register replica
 
−
| EGI
 
−
|-
 
−
| style="background-color: red;" | 136884
 
−
| t2k.org
 
−
| in progress
 
−
| top priority
 
−
| 27/08/2018
 
−
| 24/09/2018
 
−
| Data Management - generic
 
−
| lcg-cr not working for t2k vo
 
 
| EGI
 
| EGI
 
|-
 
|-
Line 260: Line 267:
 
| very urgent
 
| very urgent
 
| 23/08/2018
 
| 23/08/2018
−
| 20/09/2018
+
| 25/09/2018
 
| Other
 
| Other
 
| Cannot upload files to LFN from Storage node
 
| Cannot upload files to LFN from Storage node
 
| EGI
 
| EGI
 
|-
 
|-
−
| style="background-color: green;" | 136757
+
| style="background-color: red;" | 136701
−
| mice
+
−
| in progress
+
−
| less urgent
+
−
| 17/08/2018
+
−
| 21/08/2018
+
−
| Other
+
−
| Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ?
+
−
| EGI
+
−
|-
+
−
| style="background-color: orange;" | 136701
+
 
| lhcb
 
| lhcb
 
| waiting for reply
 
| waiting for reply
 
| very urgent
 
| very urgent
 
| 14/08/2018
 
| 14/08/2018
−
| 24/09/2018
+
| 26/09/2018
 
| File Transfer
 
| File Transfer
 
| background of transfer errors
 
| background of transfer errors
 
| WLCG
 
| WLCG
−
|-
 
−
| style="background-color: green;" | 136366
 
−
| mice
 
−
| in progress
 
−
| less urgent
 
−
| 25/07/2018
 
−
| 20/08/2018
 
−
| Local Batch System
 
−
| Remove MICE Queue from RAL T1 Batch
 
−
| EGI
 
 
|-
 
|-
 
| style="background-color: red;" | 136199
 
| style="background-color: red;" | 136199
Line 315: Line 302:
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: yellow;" | 124876
+
| style="background-color: orange;" | 124876
 
| ops
 
| ops
 
| in progress
 
| in progress
Line 349: Line 336:
 
! Scope
 
! Scope
 
|-
 
|-
−
| 137169
+
| 137294
−
| none
+
| snoplus.snolab.ca
 
| solved
 
| solved
−
| less urgent
+
| very urgent
−
| 13/09/2018
+
| 20/09/2018
−
| 14/09/2018
+
| 21/09/2018
−
| Monitoring
+
| File Access
−
| enable the nagios notifications - CVMFS
+
| No copy on tape error on Castor at RAL
 
| EGI
 
| EGI
 
|-
 
|-
−
| 137128
+
| 137267
−
| snoplus.snolab.ca
+
| atlas
 
| solved
 
| solved
 
| urgent
 
| urgent
−
| 11/09/2018
+
| 19/09/2018
−
| 14/09/2018
+
| 19/09/2018
−
| File Access
+
| Other
−
| RAL remote host name change?
+
| RAL-LCG2 transfer fail with "Communication error on send"
 +
| WLCG
 +
|-
 +
| 137151
 +
| t2k.org
 +
| solved
 +
| urgent
 +
| 12/09/2018
 +
| 24/09/2018
 +
| File Transfer
 +
| Cannot upload files to grid an register replica
 
| EGI
 
| EGI
 
|-
 
|-
−
| 136914
+
| 137136
−
| mice
+
| lhcb
 
| verified
 
| verified
−
| less urgent
+
| very urgent
−
| 28/08/2018
+
 
| 12/09/2018
 
| 12/09/2018
−
| Storage Systems
+
| 18/09/2018
−
| Lock-down of MICE Miscellaneous data on Castor
+
| File Access
−
| EGI
+
| Data transfers problem at RAL-LCG2
 +
| WLCG
 
|-
 
|-
−
| 136913
+
| 137047
−
| mice
+
| t2k.org
−
| verified
+
| solved
 
| less urgent
 
| less urgent
−
| 28/08/2018
+
| 06/09/2018
−
| 12/09/2018
+
| 20/09/2018
−
| Storage Systems
+
| File Access
−
| Lock-down of MICE RAW data
+
| Connection to RAL tape storage fails
 
| EGI
 
| EGI
 
|-
 
|-
−
| 136865
+
| 136967
 
| cms
 
| cms
 
| closed
 
| closed
 
| urgent
 
| urgent
−
| 24/08/2018
+
| 31/08/2018
−
| 11/09/2018
+
| 18/09/2018
−
| CMS_Facilities
+
−
| T1_UK_RAL squid failures and warns
+
−
| WLCG
+
−
|-
+
−
| 136859
+
−
| cms
+
−
| closed
+
−
| top priority
+
−
| 23/08/2018
+
−
| 11/09/2018
+
 
| CMS_Data Transfers
 
| CMS_Data Transfers
−
| RAL FTS server is down
+
| Phedex transfers failing from RAL to FNAL
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 136563
+
| 136942
−
| cms
+
| t2k.org
 
| closed
 
| closed
−
| urgent
+
| less urgent
−
| 06/08/2018
+
| 29/08/2018
−
| 11/09/2018
+
| 20/09/2018
−
| CMS_Data Transfers
+
| File Transfer
−
| Possibly corrupted files at RAL
+
| Copying ONLINE_AND_NEARLINE files from RAL tape storage times out
−
| WLCG
+
| EGI
−
|}
+
|-
−
<!-- **********************End Availability Report************************** ----->
+
| 136884
 +
| t2k.org
 +
| solved
 +
| top priority
 +
| 27/08/2018
 +
| 24/09/2018
 +
| Data Management - generic
 +
| lcg-cr not working for t2k vo
 +
| EGI
 +
|}<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- **********************End GGUS Tickets************************** ----->
 
<!-- **********************End GGUS Tickets************************** ----->
Line 443: Line 439:
 
! Comments
 
! Comments
 
|-
 
|-
−
| 2018-09-10
+
| 2018-09-17
−
| 100
+
 
| 100
 
| 100
 
| 100
 
| 100
 +
| style="background-color: orange;" | 96
 
| 100
 
| 100
 
| 100
 
| 100
Line 452: Line 448:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-11
+
| 2018-09-18
 
| 100
 
| 100
 
| 100
 
| 100
Line 461: Line 457:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-12
+
| 2018-09-19
 
| 100
 
| 100
 
| 100
 
| 100
Line 470: Line 466:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-13
+
| 2018-09-20
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| 98
 
 
| 100
 
| 100
 
| 100
 
| 100
Line 479: Line 475:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-14
+
| 2018-09-21
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: orange;" | 96
+
| 97
 
| 100
 
| 100
 
| 100
 
| 100
Line 488: Line 484:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-15
+
| 2018-09-22
 
| 100
 
| 100
 
| 100
 
| 100
−
| 98
+
| style="background-color: orange;" | 96
 
| 100
 
| 100
 
| 100
 
| 100
Line 497: Line 493:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-16
+
| 2018-09-23
 
| 100
 
| 100
 
| 100
 
| 100
Line 506: Line 502:
 
|  
 
|  
 
|-
 
|-
−
| 2018-09-17
+
| 2018-09-24
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: red;" | 45
 
 
| 100
 
| 100
 
| 100
 
| 100

Latest revision as of 10:02, 26 September 2018

RAL Tier1 Operations Report for 24th September 2018

Review of Issues during the week 17th August to the 24th September 2018.
  • Tier-1 have spent the last week in the process of trying to resolve the on-going LFC issues.
  • Our current understanding of the problem is that we have a corrupted LFC database. More correctly we believe a value (or values), within that database is causing an issue. We don't often see metadata showing a file-size of 18446744073709394047 bytes (yes, we think that's a yottabyte or two), and feel this may be causing some issues!
  • We have restored back to a version of the database that works with gfal-copy (i.e. doesn't throw the error people were encountering) and are now working out what's missing and corrupted between the latest working and rolled back version of the database.
  • This issue was successfully resolve with the removal of the GOCDB unscheduled down-time at 12:00BST 24/9/18.
  • A number of associated LFC tickets are also being worked on in parallel.
Current operational status and issues
  • Tier-1 encountered another IPv6 issue 17/9/18 - 23:00), however Digital Solutions were able to resolve this issue within 2 hours
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
- - - - -
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss747 Atlas atlasStripInput d1t0 Currently in intervention.
gdss753 Atlas atlasStripInput d1t0 Currently in intervention.
Limits on concurrent batch system jobs.
  • GROUP_CMS_LIMIT = 4000
  • GROUP_ATLAS_LIMIT = 8000
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
LFC 26064 No Outage 20-Sep-18 17:00:00 24-Sep-18 11:00:02 - investigating problems with the LFC - possibly in the back end database.
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137398 cms in progress urgent 26/09/2018 26/09/2018 CMS_Data Transfers Transfers failing from SPRACE to RAL - No data available WLCG
137391 atlas in progress urgent 25/09/2018 26/09/2018 Network problem UK RAL-LCG2 transfer errors with Communication error on send WLCG
137254 ops in progress less urgent 18/09/2018 25/09/2018 Operations [Rod Dashboard] Issue detected : ch.cern.LFC-Write-ops@lfc.gridpp.rl.ac.uk EGI
137195 ops in progress less urgent 14/09/2018 24/09/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
137153 t2k.org in progress urgent 12/09/2018 25/09/2018 Data Management - generic LFC entry has file size 0, preventsw registering of additional replicas EGI
136840 snoplus.snolab.ca in progress very urgent 23/08/2018 25/09/2018 Other Cannot upload files to LFN from Storage node EGI
136701 lhcb waiting for reply very urgent 14/08/2018 26/09/2018 File Transfer background of transfer errors WLCG
136199 lhcb in progress very urgent 18/07/2018 07/08/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
136028 cms in progress top priority 10/07/2018 21/09/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG
124876 ops in progress less urgent 07/11/2016 23/07/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137294 snoplus.snolab.ca solved very urgent 20/09/2018 21/09/2018 File Access No copy on tape error on Castor at RAL EGI
137267 atlas solved urgent 19/09/2018 19/09/2018 Other RAL-LCG2 transfer fail with "Communication error on send" WLCG
137151 t2k.org solved urgent 12/09/2018 24/09/2018 File Transfer Cannot upload files to grid an register replica EGI
137136 lhcb verified very urgent 12/09/2018 18/09/2018 File Access Data transfers problem at RAL-LCG2 WLCG
137047 t2k.org solved less urgent 06/09/2018 20/09/2018 File Access Connection to RAL tape storage fails EGI
136967 cms closed urgent 31/08/2018 18/09/2018 CMS_Data Transfers Phedex transfers failing from RAL to FNAL WLCG
136942 t2k.org closed less urgent 29/08/2018 20/09/2018 File Transfer Copying ONLINE_AND_NEARLINE files from RAL tape storage times out EGI
136884 t2k.org solved top priority 27/08/2018 24/09/2018 Data Management - generic lcg-cr not working for t2k vo EGI

Availability Report

Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-09-17 100 100 96 100 100 100
2018-09-18 100 100 100 100 100 100
2018-09-19 100 100 100 100 100 100
2018-09-20 100 100 100 100 100 100
2018-09-21 100 100 97 100 100 100
2018-09-22 100 100 96 100 100 100
2018-09-23 100 100 100 100 100 100
2018-09-24 100 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-08-10 100 98
2018-08-11 100 98
2018-08-12 98 99
2018-08-13 95 98
2018-08-14 100 99
2018-08-15 79 99
2018-08-16 100 99
2018-08-17 - -

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-09-24&oldid=18589"