Difference between revisions of "Tier1 Operations Report 2018-10-15"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(8 intermediate revisions by one user not shown)
Line 11: Line 11:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 8th October 2018 to the 15th October 2018.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 8th October 2018 to the 15th October 2018.
 
|}
 
|}
−
*  
+
* No major issues of note to report for this week.
 +
* Occasional repeat of a  CMS error – “500-globus_xio: System error in bind: Address already in use.” -  This is not a major issue but does impact the SAM statistics.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 22: Line 23:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
*  
+
* As an advisory Tier-1 is continuing with the current round of security patching  (CVE-2018-14634, CVE-2018-8897 – SL7 and SL6 respectively).  This involves all Tier-1 services but the risk of impact on availability is considered to be minimal and as such no declaration of service “At Risk” or “Outage” will be called.
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 41: Line 42:
 
! Comments
 
! Comments
 
|-
 
|-
−
| gdss738
+
| gdss753
−
| LHCb
+
| Atlas
−
| LHCb_FAILOVER,LHCb-Disk
+
| atlasStripInput
 
| d1t0  
 
| d1t0  
−
| Currently in intervention.
+
| Decommissioned.
 
|}
 
|}
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 177: Line 178:
 
* Internal
 
* Internal
 
** DNS servers will be rolled out within the Tier1 network.
 
** DNS servers will be rolled out within the Tier1 network.
−
 
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ************************************************************************** ----->
 
<!-- ************************************************************************** ----->
Line 200: Line 200:
 
! Subject
 
! Subject
 
! Scope
 
! Scope
−
|-
 
−
| style="background-color: green;" | 137675
 
−
| ops
 
−
| in progress
 
−
| less urgent
 
−
| 10/10/2018
 
−
| 10/10/2018
 
−
| Operations
 
−
| [Rod Dashboard] Issue detected : org.nordugrid.ARC-CE-ARIS@arc-ce04.gridpp.rl.ac.uk
 
−
| EGI
 
 
|-
 
|-
 
| style="background-color: green;" | 137650
 
| style="background-color: green;" | 137650
 
| cms
 
| cms
−
| in progress
+
| waiting for reply
 
| urgent
 
| urgent
 
| 09/10/2018
 
| 09/10/2018
−
| 09/10/2018
+
| 15/10/2018
 
| CMS_AAA WAN Access
 
| CMS_AAA WAN Access
 
| Low HC xrootd success rates at T1_UK_RAL
 
| Low HC xrootd success rates at T1_UK_RAL
Line 226: Line 216:
 
| less urgent
 
| less urgent
 
| 14/09/2018
 
| 14/09/2018
−
| 10/10/2018
+
| 15/10/2018
 
| Operations
 
| Operations
 
| [Rod Dashboard] Issues detected at RAL-LCG2
 
| [Rod Dashboard] Issues detected at RAL-LCG2
Line 256: Line 246:
 
| very urgent
 
| very urgent
 
| 18/07/2018
 
| 18/07/2018
−
| 10/10/2018
+
| 11/10/2018
 
| File Transfer
 
| File Transfer
 
| Lots of submitted transfers on RAL FTS
 
| Lots of submitted transfers on RAL FTS
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: orange;" | 124876
+
| style="background-color: red;" | 124876
 
| ops
 
| ops
 
| in progress
 
| in progress
Line 295: Line 285:
 
! Scope
 
! Scope
 
|-
 
|-
−
| style="background-color: green;" | 137619
+
| 137723
 
| cms
 
| cms
 
| solved
 
| solved
 
| urgent
 
| urgent
−
| 07/10/2018
+
| 14/10/2018
−
| 08/10/2018
+
| 15/10/2018
−
| CMS_AAA WAN Access
+
| CMS_Data Transfers
−
| T1_UK_RAL xrootd read failures
+
| PhEDEx component Agent Watchdog donw at T1_UK-RAL
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: green;" | 137565
+
| 137699
−
| atlas
+
| dteam
−
| solved
+
| verified
−
| less urgent
+
| top priority
−
| 03/10/2018
+
| 11/10/2018
−
| 05/10/2018
+
| 15/10/2018
−
| Other
+
| Monitoring
−
| failing handshake for transfers from CA-VICTORIA-WESTGRID-T2_DATADISK to UK RAL-LCG2-ECHO
+
| Test of RAL-LCG2 Alarm Ticket Handling
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: green;" | 137498
+
| style="background-color: green;" | 137675
 +
| ops
 +
| verified
 +
| less urgent
 +
| 10/10/2018
 +
| 11/10/2018
 +
| Operations
 +
| [Rod Dashboard] Issue detected : org.nordugrid.ARC-CE-ARIS@arc-ce04.gridpp.rl.ac.uk
 +
| EGI
 +
|-
 +
| 137634
 
| cms
 
| cms
 
| solved
 
| solved
 
| urgent
 
| urgent
−
| 01/10/2018
+
| 08/10/2018
−
| 05/10/2018
+
| 10/10/2018
−
| CMS_AAA WAN Access
+
| CMS_Data Transfers
−
| Xrootd FileOpenErrors in production jobs
+
| Transfers failing from T1_UK_RAL_Disk to METU
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: green;" | 137416
+
| 137619
−
| none
+
−
| verified
+
−
| top priority
+
−
| 26/09/2018
+
−
| 01/10/2018
+
−
| Other
+
−
| This TEST ALARM has been raised for testing GGUS alarm work flow after a new GGUS release.
+
−
| WLCG
+
−
|-
+
−
| style="background-color: green;" | 137398
+
 
| cms
 
| cms
 
| solved
 
| solved
 
| urgent
 
| urgent
−
| 26/09/2018
+
| 07/10/2018
−
| 03/10/2018
+
| 08/10/2018
−
| CMS_Data Transfers
+
| CMS_AAA WAN Access
−
| Transfers failing from SPRACE to RAL - No data available
+
| T1_UK_RAL xrootd read failures
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: green;" | 137294
+
| 137434
−
| snoplus.snolab.ca
+
| cms
 
| closed
 
| closed
−
| very urgent
+
| urgent
−
| 20/09/2018
+
| 26/09/2018
−
| 05/10/2018
+
| 11/10/2018
−
| File Access
+
| CMS_AAA WAN Access
−
| No copy on tape error on Castor at RAL
+
| xrootd host not reachable at T1_UK_RAL
−
| EGI
+
| WLCG
 
|-
 
|-
−
| style="background-color: green;" | 137267
+
| 137391
 
| atlas
 
| atlas
−
| closed
+
| solved
 
| urgent
 
| urgent
−
| 19/09/2018
+
| 25/09/2018
−
| 03/10/2018
+
| 10/10/2018
−
| Other
+
| Network problem
−
| RAL-LCG2 transfer fail with "Communication error on send"
+
| UK RAL-LCG2 transfer errors with Communication error on send
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| style="background-color: green;" | 137254
+
| 137151
−
| ops
+
−
| verified
+
−
| less urgent
+
−
| 18/09/2018
+
−
| 04/10/2018
+
−
| Operations
+
−
| [Rod Dashboard] Issue detected : ch.cern.LFC-Write-ops@lfc.gridpp.rl.ac.uk
+
−
| EGI
+
−
|-
+
−
| style="background-color: green;" | 137151
+
 
| t2k.org
 
| t2k.org
 
| closed
 
| closed
Line 385: Line 365:
 
| EGI
 
| EGI
 
|-
 
|-
−
| style="background-color: green;" | 137047
+
| 136757
−
| t2k.org
+
−
| closed
+
−
| less urgent
+
−
| 06/09/2018
+
−
| 04/10/2018
+
−
| File Access
+
−
| Connection to RAL tape storage fails
+
−
| EGI
+
−
|-
+
−
| style="background-color: green;" | 136840
+
−
| snoplus.snolab.ca
+
−
| solved
+
−
| very urgent
+
−
| 23/08/2018
+
−
| 03/10/2018
+
−
| Other
+
−
| Cannot upload files to LFN from Storage node
+
−
| EGI
+
−
|-
+
−
| style="background-color: green;" | 136757
+
 
| mice
 
| mice
 
| closed
 
| closed
Line 414: Line 374:
 
| Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ?
 
| Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ?
 
| EGI
 
| EGI
 +
|-
 +
| 136028
 +
| cms
 +
| closed
 +
| top priority
 +
| 10/07/2018
 +
| 12/10/2018
 +
| CMS_AAA WAN Access
 +
| Issues reading files at T1_UK_RAL_Disk
 +
| WLCG
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
Line 436: Line 406:
 
! LHCB
 
! LHCB
 
! Alice
 
! Alice
−
! OPS
 
−
! Comments
 
 
|-
 
|-
−
| 2018-10-01
+
| 2018-10-08
−
| 53
+
 
| 100
 
| 100
 
| 100
 
| 100
Line 446: Line 413:
 
| 100
 
| 100
 
| 100
 
| 100
−
|
 
 
|-
 
|-
−
| 2018-10-02
+
| 2018-10-09
−
| 91
+
−
| 99
+
 
| 100
 
| 100
 
| 100
 
| 100
 +
| 99
 
| 100
 
| 100
 
| 100
 
| 100
−
|
 
 
|-
 
|-
−
| 2018-10-03
+
| 2018-10-10
 
| 100
 
| 100
 
| 100
 
| 100
 +
| 98
 
| 100
 
| 100
 
| 100
 
| 100
−
| 100
 
−
| 100
 
−
|
 
 
|-
 
|-
−
| 2018-10-04
+
| 2018-10-11
 
| 100
 
| 100
 
| 100
 
| 100
 +
| 98
 
| 100
 
| 100
 
| 100
 
| 100
−
| 100
 
−
| 100
 
−
|
 
 
|-
 
|-
−
| 2018-10-05
+
| 2018-10-12
 
| 100
 
| 100
 
| 100
 
| 100
Line 481: Line 441:
 
| 100
 
| 100
 
| 100
 
| 100
−
| 100
 
−
|
 
 
|-
 
|-
−
| 2018-10-06
+
| 2018-10-13
 
| 100
 
| 100
 
| 100
 
| 100
Line 490: Line 448:
 
| 100
 
| 100
 
| 100
 
| 100
−
| 100
 
−
|
 
 
|-
 
|-
−
| 2018-10-07
+
| 2018-10-14
 
| 100
 
| 100
 
| 100
 
| 100
−
| 99
 
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
|
 
 
|-
 
|-
−
| 2018-10-08
+
| 2018-10-15
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
| 100
+
−
|
+
−
|-
+
−
| 2018-10-09
+
−
| 100
+
 
| 100
 
| 100
 
| 100
 
| 100
Line 518: Line 462:
 
| 100
 
| 100
 
| 100
 
| 100
−
|
 
 
|}
 
|}
  

Latest revision as of 09:55, 16 October 2018

RAL Tier1 Operations Report for 15th October 2018

Review of Issues during the week 8th October 2018 to the 15th October 2018.
  • No major issues of note to report for this week.
  • Occasional repeat of a CMS error – “500-globus_xio: System error in bind: Address already in use.” - This is not a major issue but does impact the SAM statistics.
Current operational status and issues
  • As an advisory Tier-1 is continuing with the current round of security patching (CVE-2018-14634, CVE-2018-8897 – SL7 and SL6 respectively). This involves all Tier-1 services but the risk of impact on availability is considered to be minimal and as such no declaration of service “At Risk” or “Outage” will be called.
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss753 Atlas atlasStripInput d1t0 Decommissioned.
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss747 Atlas atlasStripInput d1t0 Currently in intervention.
Limits on concurrent batch system jobs.
  • None currently enforced.
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Disk only storage will end in Castor. A new single tape-only Castor instance (called WLCGTape) is being tested. This is using generic Catsor headnodes on SL7 configured by Quattor/Aquilon with a slightly newer Castor version.
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137650 cms waiting for reply urgent 09/10/2018 15/10/2018 CMS_AAA WAN Access Low HC xrootd success rates at T1_UK_RAL WLCG
137195 ops in progress less urgent 14/09/2018 15/10/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
137153 t2k.org in progress urgent 12/09/2018 10/10/2018 Data Management - generic LFC entry has file size 0, preventsw registering of additional replicas EGI
136701 lhcb waiting for reply very urgent 14/08/2018 10/10/2018 File Transfer background of transfer errors WLCG
136199 lhcb waiting for reply very urgent 18/07/2018 11/10/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
124876 ops in progress less urgent 07/11/2016 08/10/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137723 cms solved urgent 14/10/2018 15/10/2018 CMS_Data Transfers PhEDEx component Agent Watchdog donw at T1_UK-RAL WLCG
137699 dteam verified top priority 11/10/2018 15/10/2018 Monitoring Test of RAL-LCG2 Alarm Ticket Handling WLCG
137675 ops verified less urgent 10/10/2018 11/10/2018 Operations [Rod Dashboard] Issue detected : org.nordugrid.ARC-CE-ARIS@arc-ce04.gridpp.rl.ac.uk EGI
137634 cms solved urgent 08/10/2018 10/10/2018 CMS_Data Transfers Transfers failing from T1_UK_RAL_Disk to METU WLCG
137619 cms solved urgent 07/10/2018 08/10/2018 CMS_AAA WAN Access T1_UK_RAL xrootd read failures WLCG
137434 cms closed urgent 26/09/2018 11/10/2018 CMS_AAA WAN Access xrootd host not reachable at T1_UK_RAL WLCG
137391 atlas solved urgent 25/09/2018 10/10/2018 Network problem UK RAL-LCG2 transfer errors with Communication error on send WLCG
137151 t2k.org closed urgent 12/09/2018 08/10/2018 File Transfer Cannot upload files to grid an register replica EGI
136757 mice closed less urgent 17/08/2018 09/10/2018 Other Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ? EGI
136028 cms closed top priority 10/07/2018 12/10/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG

Availability Report

Day Atlas Atlas-Echo CMS LHCB Alice
2018-10-08 100 100 100 100 100
2018-10-09 100 100 99 100 100
2018-10-10 100 100 98 100 100
2018-10-11 100 100 98 100 100
2018-10-12 100 100 100 100 100
2018-10-13 100 100 100 100 100
2018-10-14 100 100 100 100 100
2018-10-15 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-08-01 100 98
2018-08-02 100 99
2018-08-03 100 99
2018-08-04 100 99
2018-08-05 100 98
2018-08-06 100 98
2018-08-07 100 96
2018-08-08 100 96

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-10-15&oldid=18727"