Difference between revisions of "Tier1 Operations Report 2018-06-25"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(13 intermediate revisions by one user not shown)
Line 34: Line 34:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues
 
|}
 
|}
−
* gdss687 - LHCb - LHCbDst - d1t0.  Currently back in Production.
+
* gdss746 - ATLAS - atlasStripInut - d1t0.  Back in full production RW.
−
* gdss738 - LHCb - LHCbDst - d1t0.  Currently back in Production RO.
+
* gdss685 - ATLAS - atlasTape - d1t0.  Back in full production RW.
 +
* gdss776 - LHCb - LHCb_FAILOVER,LHCb-Disk - d1t0.  Currently back in production RO.
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
  
Line 45: Line 46:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues
 
|}
 
|}
−
 
+
* None.
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ************************************************************* ----->
 
<!-- ************************************************************* ----->
Line 91: Line 92:
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
|}
 
|}
−
* None
+
* No downtime scheduled in the GOCDB for next 2 weeks
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 114: Line 115:
 
** DNS servers will be rolled out within the Tier1 network.
 
** DNS servers will be rolled out within the Tier1 network.
 
* Infrastructure
 
* Infrastructure
−
** Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.
+
** Testing of power distribution boards in the R89 machine room is being scheduled for 24th – 26th July. The effect of this on our services is anticipated to be minimal.
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ************************************************************************** ----->
 
<!-- ************************************************************************** ----->
Line 137: Line 138:
 
! Subject
 
! Subject
 
! Scope
 
! Scope
−
|-
 
−
| style="background-color: green;" | 135711
 
−
| cms
 
−
| waiting for reply
 
−
| urgent
 
−
| 18/06/2018
 
−
| 19/06/2018
 
−
| CMS_Central Workflows
 
−
| T1_UK_RAL production jobs failing
 
−
| WLCG
 
 
|-
 
|-
 
| style="background-color: green;" | 135455
 
| style="background-color: green;" | 135455
Line 212: Line 203:
 
! Scope
 
! Scope
 
|-
 
|-
−
| 135661
+
| 135740
−
| atlas
+
| none
 
| solved
 
| solved
−
| less urgent
+
| urgent
−
| 14/06/2018
+
| 20/06/2018
−
| 14/06/2018
+
| 20/06/2018
−
| Databases
+
| File Access
−
| RAL-LCG2: ATLAS RAL Frontier server down
+
| CVMFS issue on RALfor  /cvmfs/dune.opensciencegrid.org/
−
| WLCG
+
−
|-
+
−
| 135367
+
−
| snoplus.snolab.ca
+
−
| solved
+
−
| less urgent
+
−
| 28/05/2018
+
−
| 13/06/2018
+
−
| Other
+
−
| Lost access to srm-snoplus.gridpp.rl.ac.uk
+
 
| EGI
 
| EGI
 
|-
 
|-
−
| 135308
+
| 135723
−
| mice
+
| lhcb
 
| solved
 
| solved
 
| top priority
 
| top priority
−
| 24/05/2018
+
| 19/06/2018
−
| 13/06/2018
+
| 19/06/2018
−
| Information System
+
| File Transfer
−
| Can't send data to RAL Castor
+
| lcgfts3 FTS server fails all transfers
−
| EGI
+
| WLCG
 
|-
 
|-
−
| 134468
+
| 135711
 +
| cms
 +
| solved
 +
| urgent
 +
| 18/06/2018
 +
| 22/06/2018
 +
| CMS_Central Workflows
 +
| T1_UK_RAL production jobs failing
 +
| WLCG
 +
|-
 +
| 135516
 
| cms
 
| cms
 
| closed
 
| closed
−
| top priority
+
| urgent
−
| 09/04/2018
+
| 06/06/2018
−
| 15/06/2018
+
| 22/06/2018
−
| CMS_AAA WAN Access
+
| CMS_Facilities
−
| Xrootd redirector not seeing some files in ECHO
+
| T1_UK_RAL WN-analysis and HammerCloud failures
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 117683
+
| 135133
−
| none
+
| cms
 
| closed
 
| closed
−
| less urgent
+
| urgent
−
| 18/11/2015
+
| 15/05/2018
−
| 15/06/2018
+
| 19/06/2018
−
| Information System
+
| CMS_Data Transfers
−
| CASTOR at RAL not publishing GLUE 2
+
| Likely corrupted File at T1_UK_RAL
 +
| WLCG
 +
|-
 +
| 134703
 +
| cms
 +
| closed
 +
| urgent
 +
| 23/04/2018
 +
| 21/06/2018
 +
| CMS_Data Transfers
 +
| Transfer failing from RAL_Disk
 +
| WLCG
 +
|-
 +
| 127597
 +
| cms
 +
| closed
 +
| urgent
 +
| 07/04/2017
 +
| 21/06/2018
 +
| File Transfer
 +
| Check networking and xrootd RAL-CERN performance
 
| EGI
 
| EGI
 
|}
 
|}
Line 292: Line 303:
 
! Comments
 
! Comments
 
|-
 
|-
−
| 2018-06-11
+
| 2018-06-18
 +
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: red;" |78
 
 
| 100
 
| 100
 
| 100
 
| 100
Line 301: Line 312:
 
|  
 
|  
 
|-
 
|-
−
| 2018-06-12
+
| 2018-06-19
−
| 100
+
| 99
−
| 100
+
| 99
−
| style="background-color: orange;" |93
+
| 99
 
| 100
 
| 100
 
| 100
 
| 100
Line 310: Line 321:
 
|  
 
|  
 
|-
 
|-
−
| 2018-06-13
+
| 2018-06-20
−
| 100
+
 
| 100
 
| 100
 
| 100
 
| 100
 +
| style="background-color: orange;" | 96
 
| 100
 
| 100
 
| 100
 
| 100
Line 319: Line 330:
 
|  
 
|  
 
|-
 
|-
−
| 2018-06-14
+
| 2018-06-21
−
| 100
+
 
| 100
 
| 100
 
| 100
 
| 100
 +
| style="background-color: orange;" | 93
 
| 100
 
| 100
 
| 100
 
| 100
Line 328: Line 339:
 
|  
 
|  
 
|-
 
|-
−
| 2018-06-15
+
| 2018-06-22
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: orange;" |92
+
| style="background-color: red;" | 85
 
| 100
 
| 100
 
| 100
 
| 100
Line 337: Line 348:
 
|  
 
|  
 
|-
 
|-
−
| 2018-06-16
+
| 2018-06-23
 
| 100
 
| 100
 
| 100
 
| 100
Line 346: Line 357:
 
|  
 
|  
 
|-
 
|-
−
| 2018-06-17
+
| 2018-06-24
 
| 100
 
| 100
 
| 100
 
| 100
Line 355: Line 366:
 
|  
 
|  
 
|-
 
|-
−
| 2018-06-18
+
| 2018-06-25
 
| 100
 
| 100
 
| 100
 
| 100
Line 381: Line 392:
 
! Day !! Atlas HC !! CMS HC !! Comment
 
! Day !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 2018/06/11 || 97 ||  style="background-color: red;" |72||  
+
| 2018/06/19 || style="background-color: red;" | 89 || 99 ||  
−
|-
+
−
| 2018/06/12 || 97 || 100||  
+
 
|-
 
|-
−
| 2018/06/13 || style="background-color: orange;" | 95    || style="background-color: red;" |77 ||  
+
| 2018/06/20 || 100 || style="background-color: red;" | 83||  
 
|-
 
|-
−
| 2018/06/14 || style="background-color: orange;" | 92    || style="background-color: red;" |45 ||  
+
| 2018/06/21 || style="background-color: orange;" | 92    || 97 ||  
 
|-
 
|-
−
| 2018/06/15 || style="background-color: orange;" | 85 || style="background-color: red;" |73 ||  
+
| 2018/06/22 || style="background-color: red;" | 85   || 100 ||  
 
|-
 
|-
−
| 2018/06/16 || style="background-color: orange;" | 95    || 100 ||  
+
| 2018/06/23 || 98 || 100 ||  
 
|-
 
|-
−
| 2018/06/17 || 97    || style="background-color: red;" |66 || 100
+
| 2018/06/24 || style="background-color: orange;" | 93    || 100 ||
 
|-
 
|-
−
| 2018/06/18 || - || - ||  
+
| 2018/06/25 || -   || - ||  
 
|-
 
|-
 
|}  
 
|}  

Latest revision as of 11:53, 27 June 2018

RAL Tier1 Operations Report for 25th June 2018

Review of Issues during the week 18th June to the 25th June 2018.
  • 12/6/18 there was an internal network problem, which took around ~3 hours to resolve. This seemed to mostly affect facilities rather than Tier-1 services. The concern is not that a piece of hardware failed, but that it doesn’t always switch seamlessly to the backup links (especially for IPv6).
  • 19/6/18 09:00 - 13:00 IPv6 was not available. Fabric attempted a route switch failed. Should have dropped to IPv4 but didn't. Need to raise this issue with Networking/Fabric to try and resolve once and for all
Current operational status and issues
  • None.
Resolved Castor Disk Server Issues
  • gdss746 - ATLAS - atlasStripInut - d1t0. Back in full production RW.
  • gdss685 - ATLAS - atlasTape - d1t0. Back in full production RW.
  • gdss776 - LHCb - LHCb_FAILOVER,LHCb-Disk - d1t0. Currently back in production RO.
Ongoing Castor Disk Server Issues
  • None.
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
  • None
Declared in the GOC DB
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Networking
    • Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
  • Infrastructure
    • Testing of power distribution boards in the R89 machine room is being scheduled for 24th – 26th July. The effect of this on our services is anticipated to be minimal.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135455 cms in progress less urgent 31/05/2018 04/06/2018 File Transfer Checksum verification at RAL EGI
135293 ops on hold less urgent 23/05/2018 04/06/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
134685 dteam in progress less urgent 23/04/2018 11/06/2018 Middleware please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 EGI
124876 ops on hold less urgent 07/11/2016 13/11/2017 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135740 none solved urgent 20/06/2018 20/06/2018 File Access CVMFS issue on RALfor /cvmfs/dune.opensciencegrid.org/ EGI
135723 lhcb solved top priority 19/06/2018 19/06/2018 File Transfer lcgfts3 FTS server fails all transfers WLCG
135711 cms solved urgent 18/06/2018 22/06/2018 CMS_Central Workflows T1_UK_RAL production jobs failing WLCG
135516 cms closed urgent 06/06/2018 22/06/2018 CMS_Facilities T1_UK_RAL WN-analysis and HammerCloud failures WLCG
135133 cms closed urgent 15/05/2018 19/06/2018 CMS_Data Transfers Likely corrupted File at T1_UK_RAL WLCG
134703 cms closed urgent 23/04/2018 21/06/2018 CMS_Data Transfers Transfer failing from RAL_Disk WLCG
127597 cms closed urgent 07/04/2017 21/06/2018 File Transfer Check networking and xrootd RAL-CERN performance EGI

Availability Report

Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-06-18 100 100 100 100 100 100
2018-06-19 99 99 99 100 100 100
2018-06-20 100 100 96 100 100 100
2018-06-21 100 100 93 100 100 100
2018-06-22 100 100 85 100 100 100
2018-06-23 100 100 100 100 100 100
2018-06-24 100 100 100 100 100 100
2018-06-25 100 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018/06/19 89 99
2018/06/20 100 83
2018/06/21 92 97
2018/06/22 85 100
2018/06/23 98 100
2018/06/24 93 100
2018/06/25 - -

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
  • Notes from meeting of 20th June:
  • LHBb - A number of drive failures have resulted in 8 files lost, 4 not on name server 4 just plain lost.
  • Issue with files appearing to be calculating a different checksum dependent on location, plain weird. Was thought to be a RAID issue but investigations have suggested this is not the case.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-06-25&oldid=17978"