Difference between revisions of "Tier1 Operations Report 2018-06-04"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(One intermediate revision by one user not shown)
Line 36: Line 36:
 
* gdss817 - LHCb - d1t0 - Currently back in production RO.
 
* gdss817 - LHCb - d1t0 - Currently back in production RO.
 
* gdss733 - LHCb - d1t0 - Currently back in production RO.
 
* gdss733 - LHCb - d1t0 - Currently back in production RO.
−
 
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
  
Line 262: Line 261:
 
| [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability
 
| [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability
 
| EGI
 
| EGI
 +
|-
 +
| 135133
 +
| cms
 +
| solved
 +
| urgent
 +
| 15/05/2018
 +
| 05/06/2018
 +
| CMS_Data Transfers
 +
| Likely corrupted File at T1_UK_RAL
 +
| WLCG
 
|-
 
|-
 
| 134468
 
| 134468
Line 292: Line 301:
 
| CASTOR at RAL not publishing GLUE 2
 
| CASTOR at RAL not publishing GLUE 2
 
| EGI
 
| EGI
−
|}<!-- **********************End Availability Report************************** ----->
+
|}
 +
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- **********************End GGUS Tickets************************** ----->
 
<!-- **********************End GGUS Tickets************************** ----->

Latest revision as of 10:35, 6 June 2018

RAL Tier1 Operations Report for 4th June 2018

Review of Issues during the week 28th May to the 4th June 2018.
  • No incidents(major or minor), have been flagged during this reporting period.
Current operational status and issues
  • For the avoidance of doubt regards confusion over the status of both the BDii and Castor services. Tier-1 have NOT stopped or decommissioned these services. What has changed is that the Castor information stopped getting published in the BDii. This was due to a failure of the machine running the Castor Information Provide (CIP). The CIP was a legacy part of Castor (It didn't support GLUE2 schema), and was due to be decommissioned, the hardware failure expedited this issue. Although we admit this is not ideal, we decided against bringing back a service simply to then inform people we would shortly remove it.
Resolved Castor Disk Server Issues
  • gdss801 - cmsDisk - d1t0. Currently back in production RO.
  • gdss817 - LHCb - d1t0 - Currently back in production RO.
  • gdss733 - LHCb - d1t0 - Currently back in production RO.
Ongoing Castor Disk Server Issues
  • None.
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
  • None
Declared in the GOC DB
  • None
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Networking
    • Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
  • Infrastructure
    • Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.
Open

GGUS Tickets (Snapshot taken during morning of the report).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135516 cms in progress urgent 06/06/2018 06/06/2018 CMS_Facilities T1_UK_RAL WN-analysis and HammerCloud failures WLCG
135455 cms in progress less urgent 31/05/2018 04/06/2018 File Transfer Checksum verification at RAL EGI
135367 snoplus.snolab.ca in progress less urgent 28/05/2018 05/06/2018 Other Lost access to srm-snoplus.gridpp.rl.ac.uk EGI
135308 mice in progress top priority 24/05/2018 01/06/2018 Information System Can't send data to RAL Castor EGI
135293 ops on hold less urgent 23/05/2018 04/06/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
134703 cms in progress urgent 23/04/2018 25/05/2018 CMS_Data Transfers Transfer failing from RAL_Disk WLCG
134685 dteam in progress less urgent 23/04/2018 02/05/2018 Middleware please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 EGI
127597 cms on hold urgent 07/04/2017 30/04/2018 File Transfer Check networking and xrootd RAL-CERN performance EGI
124876 ops on hold less urgent 07/11/2016 13/11/2017 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135342 ops unsolved less urgent 27/05/2018 31/05/2018 Operations [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability EGI
135133 cms solved urgent 15/05/2018 05/06/2018 CMS_Data Transfers Likely corrupted File at T1_UK_RAL WLCG
134468 cms solved top priority 09/04/2018 01/06/2018 CMS_AAA WAN Access Xrootd redirector not seeing some files in ECHO WLCG
133992 atlas verified less urgent 12/03/2018 04/06/2018 File Transfer RAL-LCG2-ECHO: No such file or directory EGI
117683 none unsolved less urgent 18/11/2015 31/05/2018 Information System CASTOR at RAL not publishing GLUE 2 EGI

Availability Report

Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-05-28 100 100 100 100 100 0
2018-05-29 100 100 100 100 100 0
2018-05-30 98 100 100 100 85 0
2018-05-31 100 100 100 100 100 0
2018-06-01 100 100 99 100 100 0
2018-06-02 98 100 97 100 100 0
2018-06-03 100 100 99 100 100 0
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018/05/29 97 99
2018/05/30 95 100
2018/05/31 97 71
2018/06/01 96 98
2018/06/02 97 98
2018/06/03 98 98
2018/06/04 95 0

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
  • None yet
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-06-04&oldid=17789"