Difference between revisions of "Tier1 Operations Report 2018-01-17"

From GridPP Wiki
Jump to: navigation, search
(→‎)
Line 9: Line 9:
 
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;"
 
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;"
 
|-
 
|-
−
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 11th January 2018 to 17th January 2018
+
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 3rd January 2018 to 10th January 2018
 
|}
 
|}
−
 
+
*
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 33: Line 33:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues
 
|}
 
|}
−
* gdss736 (LHCb - D1T0) - Has been rebuilt and returned to production
+
* gdss736 (LHCb) – rebuilt and back in production (RO)
−
* gdss756 (CMS - D1T0) - Disk server crashed double disk failure port 24,port 28, port 3 and port 8
+
* gdss728 (CMS) –  rebuilt and back in production (RW)
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
  
Line 44: Line 44:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues
 
|}
 
|}
−
* None
+
* gdss717 – cmsTape – multiple drive failure
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ************************************************************* ----->
 
<!-- ************************************************************* ----->
Line 78: Line 78:
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report.
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report.
 
|}
 
|}
−
No downtime scheduled in the GOCDB between 2017-12-20 and 2018-01-03
+
* None
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 99: Line 99:
 
! Reason
 
! Reason
 
|-
 
|-
−
| WMS Service: lcglb01, lcglb02, lcgwms04, lcgwms05  
+
|srm-alice.gridpp.rl.ac.uk, srm-atlas.gridpp.rl.ac.uk, srm-biomed.gridpp.rl.ac.uk, srm-cert.gridpp.rl.ac.uk, srm-cms-disk.gridpp.rl.ac.uk, srm-cms.gridpp.rl.ac.uk, srm-dteam.gridpp.rl.ac.uk, srm-ilc.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-mice.gridpp.rl.ac.uk, srm-minos.gridpp.rl.ac.uk, srm-na62.gridpp.rl.ac.uk, srm-pheno.gridpp.rl.ac.uk, srm-preprod.gridpp.rl.ac.uk, srm-snoplus.gridpp.rl.ac.uk, srm-solid.gridpp.rl.ac.uk, srm-t2k.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 17/01/2018 10:00
 +
| 17/01/2018 13:00
 +
| 3 hours
 +
|Outage of Castor Storage to apply Security patches.
 +
|-
 +
|lcglb01.gridpp.rl.ac.uk, lcglb02.gridpp.rl.ac.uk, lcgwms04.gridpp.rl.ac.uk, lcgwms05.gridpp.rl.ac.uk,
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
Line 105: Line 113:
 
| 19/01/2018 12:00
 
| 19/01/2018 12:00
 
| 7 days, 2 hours
 
| 7 days, 2 hours
−
| WMS Decommissioning RAL Tier1
+
|WMS Decommissioning RAL Tier1
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 159: Line 167:
 
! Subject
 
! Subject
 
|-
 
|-
−
| style="background-color: green;" | 132748
+
| style="background-color: green;" | 132844
 +
| TEAM
 +
| atlas
 +
| RAL-LCG2
 +
| NGI_UK assign to:lcg-support@gridpp.rl.ac.uk
 +
| waiting for reply
 +
| urgent
 +
| 2018-01-14 02:10:00
 +
| 2018-01-16 16:51:00
 +
| Storage Systems
 +
| UK RAL-LCG2 DATADISK transfer errors "DESTINATION OVERWRITE srm-ifce err:"
 +
|-
 +
| style="background-color: green;" | 132830
 
| USER
 
| USER
−
| ops
+
| cms
 
| RAL-LCG2
 
| RAL-LCG2
 
| NGI_UK
 
| NGI_UK
 
| in progress
 
| in progress
−
| less urgent
+
| very urgent
−
| 2018-01-08 14:45:00
+
| 2018-01-12 11:11:00
−
| 2018-01-09 09:58:00
+
| 2018-01-16 16:36:00
−
| Operations
+
| CMS_AAA WAN Access
−
| [Rod Dashboard] Issues detected at RAL-LCG2
+
| Reading issues T1_UK_RAL
 +
|-
 +
| style="background-color: green;" | 132802
 +
| USER
 +
| cms
 +
| RAL-LCG2
 +
| NGI_UK
 +
| in progress
 +
| urgent
 +
| 2018-01-11 10:30:00
 +
| 2018-01-15 11:26:00
 +
| CMS_AAA WAN Access
 +
| Low HC xrootd success rates at T1_UK_RAL
 
|-
 
|-
 
| style="background-color: green;" | 132712
 
| style="background-color: green;" | 132712
Line 175: Line 207:
 
| other
 
| other
 
| RAL-LCG2
 
| RAL-LCG2
−
| NGI_UK assign to:lcg-support@gridpp.rl.ac.uk
+
| NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:Kashif.Mohammad@PHYSICS.OX.AC.UK
 
| in progress
 
| in progress
 
| less urgent
 
| less urgent
 
| 2018-01-04 16:22:00
 
| 2018-01-04 16:22:00
−
| 2018-01-08 13:35:00
+
| 2018-01-16 15:01:00
 
| Other
 
| Other
 
| support for the hyperk VO (RAL-LCG2)
 
| support for the hyperk VO (RAL-LCG2)
 
|-
 
|-
−
| style="background-color: green;" | 132589
+
| style="background-color: yellow;" | 132589
 
| TEAM
 
| TEAM
 
| lhcb
 
| lhcb
Line 191: Line 223:
 
| very urgent
 
| very urgent
 
| 2017-12-21 06:45:00
 
| 2017-12-21 06:45:00
−
| 2018-01-03 15:21:00
+
| 2018-01-10 11:07:00
 
| Local Batch System
 
| Local Batch System
 
| Killed pilots at RAL
 
| Killed pilots at RAL
Line 203: Line 235:
 
| less urgent
 
| less urgent
 
| 2017-11-13 14:42:00
 
| 2017-11-13 14:42:00
−
| 2017-12-01 19:30:00
+
| 2018-01-12 14:18:00
 
| Storage Systems
 
| Storage Systems
 
| Extremely long download times for T2K files on tape at RAL
 
| Extremely long download times for T2K files on tape at RAL
Line 260: Line 292:
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas Echo !!  Comment
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas Echo !!  Comment
 
|-
 
|-
−
| 03/01/18 || 100 || 100 || 100 || 100 || 100 || 100 ||  
+
| 10/01/18 || 100 || 100 || 100 || 100 || 100 || 100 ||  
 
|-
 
|-
−
| 04/01/18 || 100 || 100 || 100 || 100 || 100 || 100 ||   
+
| 11/01/18 || 100 || 100 || 100 || 100 || style="background-color: yellow;" |98|| 100 ||  gdss736 (LHCb) failed
 
|-
 
|-
−
| 05/01/18 || 100 || 100 || 100 || 100 || 100|| 100 ||  
+
| 12/01/18 || 100 || 100 || 100 || 100 || 100|| 100 ||  
 
|-
 
|-
−
| 06/01/18 || 100 || 100 || 100 || 100 || 100  || 100 ||  
+
| 13/01/18 || 100 || 100 || 100 || 100 || 100  || 100 ||  
 
|-
 
|-
−
| 07/01/18 || 100 || 100 || 100 || 100 || 100 || 100  ||  
+
| 14/01/18 || 100 || style="background-color: yellow;" | 96 || style="background-color: yellow;" | 53 || 100 || 100 || 100  ||  
 
|-
 
|-
−
| 08/01/18 || 100 || 100 || 100  || 100 || 100 || 100 ||  
+
| 15/01/18 || 100 || 100 || style="background-color: yellow;" |74|| 100 || 100 || 100 ||  
 
|-
 
|-
−
| 09/01/18 || 100 || 100 || 100  || 100 || 100 || 100 ||  
+
| 16/01/18 || 100 || 100 || style="background-color: yellow;" |92|| 100 || 100 || 100 ||  
 
|}
 
|}
  
Line 288: Line 320:
 
! Day !! Atlas HC !! CMS HC !! Comment
 
! Day !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 03/01/18 || 100 || 100 ||   
+
| 10/01/18 || 100 || 100 ||   
 
|-
 
|-
−
| 04/01/18 || 100 || 100 ||  
+
| 11/01/18 || 100 || 100 ||  
 
|-
 
|-
−
| 05/01/18 || 100 || 100 ||  
+
| 12/01/18 || 100 || 100 ||  
 
|-
 
|-
−
| 06/01/18 || 100 || 100 ||  
+
| 13/01/18 || 100 || 100 ||  
 
|-
 
|-
−
| 07/01/18 || 100 || 100 ||  
+
| 14/01/18 || 100 || 100 ||  
 
|-
 
|-
−
| 08/01/18 || 100 || 100 ||  
+
| 15/01/18 || 100 || style="background-color: green;" | 99 ||  
 
|-
 
|-
−
| 09/01/18 || 100 || 100 ||  
+
| 16/01/18 || style="background-color: yellow;" | 85 || style="background-color: green;" | 99 ||
 
|}
 
|}
  

Revision as of 11:08, 17 January 2018

RAL Tier1 Operations Report for 17th January 2018

Review of Issues during the week 3rd January 2018 to 10th January 2018
Current operational status and issues
  • Ongoing security patching.
Resolved Castor Disk Server Issues
  • gdss736 (LHCb) – rebuilt and back in production (RO)
  • gdss728 (CMS) – rebuilt and back in production (RW)
Ongoing Castor Disk Server Issues
  • gdss717 – cmsTape – multiple drive failure
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • The termination of the WMS service had been announced - with a drain (i.e. not accepting new jobs) planned to start on 1st February. However, as the WMS is not being used by VOs at the moment and security patches need to be applied urgently the drain was brought forward and started this morning (10th Jan).
  • Other security patching done or underway.
Entries in GOC DB starting since the last report.
  • None
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
srm-alice.gridpp.rl.ac.uk, srm-atlas.gridpp.rl.ac.uk, srm-biomed.gridpp.rl.ac.uk, srm-cert.gridpp.rl.ac.uk, srm-cms-disk.gridpp.rl.ac.uk, srm-cms.gridpp.rl.ac.uk, srm-dteam.gridpp.rl.ac.uk, srm-ilc.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-mice.gridpp.rl.ac.uk, srm-minos.gridpp.rl.ac.uk, srm-na62.gridpp.rl.ac.uk, srm-pheno.gridpp.rl.ac.uk, srm-preprod.gridpp.rl.ac.uk, srm-snoplus.gridpp.rl.ac.uk, srm-solid.gridpp.rl.ac.uk, srm-t2k.gridpp.rl.ac.uk, SCHEDULED OUTAGE 17/01/2018 10:00 17/01/2018 13:00 3 hours Outage of Castor Storage to apply Security patches.
lcglb01.gridpp.rl.ac.uk, lcglb02.gridpp.rl.ac.uk, lcgwms04.gridpp.rl.ac.uk, lcgwms05.gridpp.rl.ac.uk, SCHEDULED OUTAGE 12/01/2018 10:00 19/01/2018 12:00 7 days, 2 hours WMS Decommissioning RAL Tier1
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Ongoing or Pending - but not yet formally announced:

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Echo:
    • Update to next CEPH version ("Luminous").
  • Networking
    • Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
  • Services
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
    • Infrastructure
  • Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.
Open GGUS Tickets (Snapshot during morning of meeting)
Ticket-ID Type VO Notified Site Resp. Unit Status Priority Creation Last Update ToI Subject
132844 TEAM atlas RAL-LCG2 NGI_UK assign to:lcg-support@gridpp.rl.ac.uk waiting for reply urgent 2018-01-14 02:10:00 2018-01-16 16:51:00 Storage Systems UK RAL-LCG2 DATADISK transfer errors "DESTINATION OVERWRITE srm-ifce err:"
132830 USER cms RAL-LCG2 NGI_UK in progress very urgent 2018-01-12 11:11:00 2018-01-16 16:36:00 CMS_AAA WAN Access Reading issues T1_UK_RAL
132802 USER cms RAL-LCG2 NGI_UK in progress urgent 2018-01-11 10:30:00 2018-01-15 11:26:00 CMS_AAA WAN Access Low HC xrootd success rates at T1_UK_RAL
132712 USER other RAL-LCG2 NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:Kashif.Mohammad@PHYSICS.OX.AC.UK in progress less urgent 2018-01-04 16:22:00 2018-01-16 15:01:00 Other support for the hyperk VO (RAL-LCG2)
132589 TEAM lhcb RAL-LCG2 NGI_UK in progress very urgent 2017-12-21 06:45:00 2018-01-10 11:07:00 Local Batch System Killed pilots at RAL
131815 USER t2k.org RAL-LCG2 NGI_UK in progress less urgent 2017-11-13 14:42:00 2018-01-12 14:18:00 Storage Systems Extremely long download times for T2K files on tape at RAL
127597 USER cms RAL-LCG2 NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:sexton@fnal.gov on hold urgent 2017-04-07 10:34:00 2017-10-05 09:14:00 File Transfer Check networking and xrootd RAL-CERN performance
124876 USER ops RAL-LCG2 NGI_UK assign to:lcg-support@gridpp.rl.ac.uk on hold less urgent 2016-11-07 12:06:00 2017-11-13 16:55:00 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
117683 USER none RAL-LCG2 NGI_UK assign to:lcg-support@gridpp.rl.ac.uk on hold less urgent 2015-11-18 11:36:00 2018-01-03 15:26:00 Information System CASTOR at RAL not publishing GLUE 2
Availability Report
Day OPS Alice Atlas CMS LHCb Atlas Echo Comment
10/01/18 100 100 100 100 100 100
11/01/18 100 100 100 100 98 100 gdss736 (LHCb) failed
12/01/18 100 100 100 100 100 100
13/01/18 100 100 100 100 100 100
14/01/18 100 96 53 100 100 100
15/01/18 100 100 74 100 100 100
16/01/18 100 100 92 100 100 100
Hammercloud Test Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_UCORE, Template 841); CMS HC = CMS HammerCloud

Day Atlas HC CMS HC Comment
10/01/18 100 100
11/01/18 100 100
12/01/18 100 100
13/01/18 100 100
14/01/18 100 100
15/01/18 100 99
16/01/18 85 99


Notes from Meeting.
  • None yet.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-01-17&oldid=16580"