Difference between revisions of "Tier1 Operations Report 2018-09-17"

From GridPP Wiki
Jump to: navigation, search
(→‎RAL Tier1 Operations Report for 27th August 2018)
(→‎)
 
(12 intermediate revisions by one user not shown)
Line 11: Line 11:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 20th August to the 17th September 2018.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 20th August to the 17th September 2018.
 
|}
 
|}
−
* Two major outages of IPv6 this week which impacted GODDB and FTS (primarily effecting ATLAS). The first started Thursday evening (~17:30 23/8/18), but was not resolved until ~22:00 24/8/18.  The same fault happened again at ~15:20 25/8/18 and was finally resolved ~22:00 that evening. DS identified a problem on router5 where the IPv6 routes were not being advertised properly and has added a static route to fix the issue.
+
* There has been improvements in ATLAS and their fair-share usage. This is ongoing work with the aim of attaining further gains.
 +
* There are currently a worrying number of GGUS tickets relating to the LFC. This is currently priority tasking for Tier-1 at this time.
 +
 
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 22: Line 24:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
* NTR
+
* Tier-1 encountered another IPv6 issue 17/9/18 - 23:00), however Digital Solutions were able to resolve this issue within 2 hours
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 41: Line 43:
 
! Comments
 
! Comments
 
|-
 
|-
−
| gdss771
+
| -
−
| LHCb
+
| -
−
| lhcbDst
+
| -
−
| d1t0
+
| -
−
| Back in production RO
+
| -
−
|-
+
 
−
| gdss732
+
−
| LHCb
+
−
| lhcbDst
+
−
| d1t0
+
−
| Back in production RO
+
 
|}
 
|}
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 73: Line 70:
 
| Atlas  
 
| Atlas  
 
| atlasStripInput
 
| atlasStripInput
−
| d1t0
 
−
| Currently in intervention.
 
−
|-
 
−
| gdss738
 
−
| LHCb
 
−
| lhcbDst
 
 
| d1t0  
 
| d1t0  
 
| Currently in intervention.
 
| Currently in intervention.
Line 127: Line 118:
 
! Reason
 
! Reason
 
|-
 
|-
−
| 25905
 
−
| FTS
 
−
| UNSCHEDULED
 
−
| OUTAGE
 
−
| 24/8/18 14:30
 
−
| 24/8/18 18:45
 
 
| -
 
| -
−
| IPv6 outage impacting FTS services
 
−
|-
 
−
| 25912
 
−
| GOCDB
 
−
| UNSCHEDULED
 
−
| OUTAGE
 
−
| 25/8/18 18:50
 
−
| 25/8/18 19:46
 
 
| -
 
| -
−
| Ongoing IPv6 outage impacting GOCDB
+
| -
 +
| -
 +
| -
 +
| -
 +
| -
 +
| -
 +
 
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 222: Line 205:
 
! Scope
 
! Scope
 
|-
 
|-
−
| style="background-color: green;" | 136942
+
| style="background-color: green;" | 137195
−
| t2k.org
+
| ops
 
| in progress
 
| in progress
 
| less urgent
 
| less urgent
−
| 29/08/2018
+
| 14/09/2018
−
| 30/08/2018
+
| 14/09/2018
 +
| Operations
 +
| [Rod Dashboard] Issues detected at RAL-LCG2
 +
| EGI
 +
|-
 +
| style="background-color: green;" | 137153
 +
| t2k.org
 +
| in progress
 +
| urgent
 +
| 12/09/2018
 +
| 13/09/2018
 +
| Data Management - generic
 +
| LFC entry has file size 0, preventsw registering of additional replicas
 +
| EGI
 +
|-
 +
| style="background-color: green;" | 137151
 +
| t2k.org
 +
| waiting for reply
 +
| urgent
 +
| 12/09/2018
 +
| 14/09/2018
 
| File Transfer
 
| File Transfer
−
| Copying ONLINE_AND_NEARLINE files from RAL tape storage times out
+
| Cannot upload files to grid an register replica
 +
| EGI
 +
|-
 +
| style="background-color: green;" | 137136
 +
| lhcb
 +
| in progress
 +
| very urgent
 +
| 12/09/2018
 +
| 14/09/2018
 +
| File Access
 +
| Data transfers problem at RAL-LCG2
 +
| WLCG
 +
|-
 +
| style="background-color: green;" | 137047
 +
| t2k.org
 +
| in progress
 +
| less urgent
 +
| 06/09/2018
 +
| 11/09/2018
 +
| File Access
 +
| Connection to RAL tape storage fails
 
| EGI
 
| EGI
 
|-
 
|-
−
| style="background-color: green;" | 136884
+
| style="background-color: orange;" | 136884
 
| t2k.org
 
| t2k.org
 
| in progress
 
| in progress
 
| top priority
 
| top priority
 
| 27/08/2018
 
| 27/08/2018
−
| 29/08/2018
+
| 16/09/2018
 
| Data Management - generic
 
| Data Management - generic
 
| lcg-cr not working for t2k vo
 
| lcg-cr not working for t2k vo
Line 245: Line 268:
 
| snoplus.snolab.ca
 
| snoplus.snolab.ca
 
| in progress
 
| in progress
−
| urgent
+
| very urgent
 
| 23/08/2018
 
| 23/08/2018
−
| 29/08/2018
+
| 15/09/2018
 
| Other
 
| Other
 
| Cannot upload files to LFN from Storage node
 
| Cannot upload files to LFN from Storage node
Line 262: Line 285:
 
| EGI
 
| EGI
 
|-
 
|-
−
| style="background-color: green;" | 136701
+
| style="background-color: orange;" | 136701
 
| lhcb
 
| lhcb
 
| in progress
 
| in progress
 
| very urgent
 
| very urgent
 
| 14/08/2018
 
| 14/08/2018
−
| 24/08/2018
+
| 05/09/2018
 
| File Transfer
 
| File Transfer
 
| background of transfer errors
 
| background of transfer errors
Line 297: Line 320:
 
| top priority
 
| top priority
 
| 10/07/2018
 
| 10/07/2018
−
| 29/08/2018
+
| 14/09/2018
 
| CMS_AAA WAN Access
 
| CMS_AAA WAN Access
 
| Issues reading files at T1_UK_RAL_Disk
 
| Issues reading files at T1_UK_RAL_Disk
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: green;" | 124876
+
| style="background-color: yellow;" | 124876
 
| ops
 
| ops
 
| in progress
 
| in progress
Line 336: Line 359:
 
! Scope
 
! Scope
 
|-
 
|-
−
| 136798
+
| 137169
−
| lhcb
+
| none
−
| verified
+
| solved
−
| very urgent
+
| less urgent
−
| 21/08/2018
+
| 13/09/2018
−
| 21/08/2018
+
| 14/09/2018
−
| File Transfer
+
| Monitoring
−
| the Request Handler does not allow forcing of the request UUID
+
| enable the nagios notifications - CVMFS
−
| WLCG
+
| EGI
 
|-
 
|-
−
| 136665
+
| 137128
−
| cms
+
| snoplus.snolab.ca
 
| solved
 
| solved
 
| urgent
 
| urgent
−
| 11/08/2018
+
| 11/09/2018
−
| 21/08/2018
+
| 14/09/2018
−
| CMS_SAM tests
+
| File Access
−
| T1_UK_RAL is down > 12h
+
| RAL remote host name change?
−
| WLCG
+
| EGI
 
|-
 
|-
−
| 136358
+
| 136914
 +
| mice
 +
| verified
 +
| less urgent
 +
| 28/08/2018
 +
| 12/09/2018
 +
| Storage Systems
 +
| Lock-down of MICE Miscellaneous data on Castor
 +
| EGI
 +
|-
 +
| 136913
 +
| mice
 +
| verified
 +
| less urgent
 +
| 28/08/2018
 +
| 12/09/2018
 +
| Storage Systems
 +
| Lock-down of MICE RAW data
 +
| EGI
 +
|-
 +
| 136865
 
| cms
 
| cms
−
| solved
+
| closed
 
| urgent
 
| urgent
−
| 25/07/2018
+
| 24/08/2018
−
| 21/08/2018
+
| 11/09/2018
 
| CMS_Facilities
 
| CMS_Facilities
−
| T1_UK_RAL WN-xrootd-access failure
+
| T1_UK_RAL squid failures and warns
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 134685
+
| 136859
−
| dteam
+
| cms
 
| closed
 
| closed
−
| less urgent
+
| top priority
−
| 23/04/2018
+
| 23/08/2018
−
| 22/08/2018
+
| 11/09/2018
−
| Middleware
+
| CMS_Data Transfers
−
| please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7
+
| RAL FTS server is down
−
| EGI
+
| WLCG
 +
|-
 +
| 136563
 +
| cms
 +
| closed
 +
| urgent
 +
| 06/08/2018
 +
| 11/09/2018
 +
| CMS_Data Transfers
 +
| Possibly corrupted files at RAL
 +
| WLCG
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
Line 388: Line 441:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" |  
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" |  
 
Availability Report
 
Availability Report
−
|}
 
−
{| border=1 align=center
 
−
| Target Availability for each site is 97.0%
 
−
| style="background-color: Red;" | Red <90%
 
−
| style="background-color: orange;" | Orange <97%
 
 
|}
 
|}
 
{| border=1 align=center
 
{| border=1 align=center
Line 405: Line 453:
 
! Comments
 
! Comments
 
|-
 
|-
−
| 2018-08-20
+
| 2018-09-10
 
| 100
 
| 100
 
| 100
 
| 100
Line 414: Line 462:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-21
+
| 2018-09-11
 
| 100
 
| 100
 
| 100
 
| 100
Line 423: Line 471:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-22
+
| 2018-09-12
 
| 100
 
| 100
 
| 100
 
| 100
Line 432: Line 480:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-23
+
| 2018-09-13
 
| 100
 
| 100
 
| 100
 
| 100
Line 441: Line 489:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-24
+
| 2018-09-14
 
| 100
 
| 100
 
| 100
 
| 100
−
| 99
+
| style="background-color: orange;" | 96
 
| 100
 
| 100
 
| 100
 
| 100
Line 450: Line 498:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-25
+
| 2018-09-15
−
| 100
+
 
| 100
 
| 100
 
| 100
 
| 100
 +
| 98
 
| 100
 
| 100
 
| 100
 
| 100
Line 459: Line 507:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-26
+
| 2018-09-16
 
| 100
 
| 100
 
| 100
 
| 100
Line 468: Line 516:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-27
+
| 2018-09-17
−
| 100
+
 
| 100
 
| 100
 
| 100
 
| 100
 +
| style="background-color: red;" | 45
 
| 100
 
| 100
 
| 100
 
| 100
Line 494: Line 542:
 
! Day !! Atlas HC !! CMS HC !! Comment
 
! Day !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 2018-08-13 || 0 ||  0 ||   
+
| 2018-08-10 || 100 ||  98 ||   
 
|-
 
|-
−
| 2018-08-14 || 0 || 0 ||  
+
| 2018-08-11 || 100 || 98 ||  
 
|-
 
|-
−
| 2018-08-15 || 0 || 0 ||  
+
| 2018-08-12 || 98 || 99 ||  
 
|-
 
|-
−
| 2018-08-16 || 0 || 0 ||  
+
| 2018-08-13 || style="background-color: orange;" | 95 || 98 ||  
 
|-
 
|-
−
| 2018-08-17 || 76 || 60||  
+
| 2018-08-14 || 100 || 99||  
 
|-
 
|-
−
| 2018-08-18 || 100 || 100||  
+
| 2018-08-15 || style="background-color: red;" | 79 || 99 ||  
 
|-
 
|-
−
| 2018-08-19 || 100 || 100 ||  
+
| 2018-08-16 || 100 || 99 ||  
 
|-
 
|-
−
| 2018-08-20 || 100 || 100 ||  
+
| 2018-08-17 || - || - ||  
 
|-
 
|-
 
|}  
 
|}  
Line 522: Line 570:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
|}
 
|}
−
* The recent problems with Echo were discussed. A summary is also in the Operations report. Several points:
+
*
−
* Current situation is that we have “production” access running
+
−
* Atlas and CMS have upper limits on batch jobs about equal to pledge.
+
−
* The additional memory for the Dell storage nodes has arrived. This will be added – at least initially in a rolling upgrade. However, it is expected this may take up to something like 2 weeks. (At which point we expect to be able to give full access to Echo).  Discussions on the best way to do the memory upgrades is ongoing.
+
−
* Before the problem we had noted that the LHCb files came from Castor in a “bursty” way. It was suggested last week that we limit the FTS to smooth out this burstiness.
+
−
* When discussing GGUS tickets: LHCb are seeing some file transfer failures between worker nodes and Castor. This to be escalated.
+
−
* The requirement for us to be able to easily contact all users of Echo in the event of a problem was noted.
+

Latest revision as of 09:54, 18 September 2018

RAL Tier1 Operations Report for 17th September 2018

Review of Issues during the week 20th August to the 17th September 2018.
  • There has been improvements in ATLAS and their fair-share usage. This is ongoing work with the aim of attaining further gains.
  • There are currently a worrying number of GGUS tickets relating to the LFC. This is currently priority tasking for Tier-1 at this time.


Current operational status and issues
  • Tier-1 encountered another IPv6 issue 17/9/18 - 23:00), however Digital Solutions were able to resolve this issue within 2 hours
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
- - - - -
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss747 Atlas atlasStripInput d1t0 Currently in intervention.
Limits on concurrent batch system jobs.
  • GROUP_CMS_LIMIT = 4000
  • GROUP_ATLAS_LIMIT = 8000
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137195 ops in progress less urgent 14/09/2018 14/09/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
137153 t2k.org in progress urgent 12/09/2018 13/09/2018 Data Management - generic LFC entry has file size 0, preventsw registering of additional replicas EGI
137151 t2k.org waiting for reply urgent 12/09/2018 14/09/2018 File Transfer Cannot upload files to grid an register replica EGI
137136 lhcb in progress very urgent 12/09/2018 14/09/2018 File Access Data transfers problem at RAL-LCG2 WLCG
137047 t2k.org in progress less urgent 06/09/2018 11/09/2018 File Access Connection to RAL tape storage fails EGI
136884 t2k.org in progress top priority 27/08/2018 16/09/2018 Data Management - generic lcg-cr not working for t2k vo EGI
136840 snoplus.snolab.ca in progress very urgent 23/08/2018 15/09/2018 Other Cannot upload files to LFN from Storage node EGI
136757 mice in progress less urgent 17/08/2018 21/08/2018 Other Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ? EGI
136701 lhcb in progress very urgent 14/08/2018 05/09/2018 File Transfer background of transfer errors WLCG
136366 mice in progress less urgent 25/07/2018 20/08/2018 Local Batch System Remove MICE Queue from RAL T1 Batch EGI
136199 lhcb in progress very urgent 18/07/2018 07/08/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
136028 cms in progress top priority 10/07/2018 14/09/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG
124876 ops in progress less urgent 07/11/2016 23/07/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137169 none solved less urgent 13/09/2018 14/09/2018 Monitoring enable the nagios notifications - CVMFS EGI
137128 snoplus.snolab.ca solved urgent 11/09/2018 14/09/2018 File Access RAL remote host name change? EGI
136914 mice verified less urgent 28/08/2018 12/09/2018 Storage Systems Lock-down of MICE Miscellaneous data on Castor EGI
136913 mice verified less urgent 28/08/2018 12/09/2018 Storage Systems Lock-down of MICE RAW data EGI
136865 cms closed urgent 24/08/2018 11/09/2018 CMS_Facilities T1_UK_RAL squid failures and warns WLCG
136859 cms closed top priority 23/08/2018 11/09/2018 CMS_Data Transfers RAL FTS server is down WLCG
136563 cms closed urgent 06/08/2018 11/09/2018 CMS_Data Transfers Possibly corrupted files at RAL WLCG

Availability Report

Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-09-10 100 100 100 100 100 100
2018-09-11 100 100 100 100 100 100
2018-09-12 100 100 100 100 100 100
2018-09-13 100 100 98 100 100 100
2018-09-14 100 100 96 100 100 100
2018-09-15 100 100 98 100 100 100
2018-09-16 100 100 100 100 100 100
2018-09-17 100 100 45 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-08-10 100 98
2018-08-11 100 98
2018-08-12 98 99
2018-08-13 95 98
2018-08-14 100 99
2018-08-15 79 99
2018-08-16 100 99
2018-08-17 - -

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-09-17&oldid=18535"