Tier1 Operations Report 2018-07-30

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 30th July 2018

Review of Issues during the week 23rd July to the 30th July 2018.
  • Every circuit breaker located in Tier-1's R89 building was tested between Tuesday 24th - Thursday 26th July. This testing period was completed successfully. The majority of racks were powered by dual PDUs which were on different circuit breakers and so theoretically, nothing should have gone down. In reality 23 PDUs failed over the 3 day period. This was within the expected number of failures although it was at the highest end of our estimates. We had sufficient replacement PDUs.
  • Impact to Tier-1:
    • 2 Production Castor disk servers went down with temporary lost of access to some file access.
    • Castor transfer manager went down for ATLAS, lost of ability to schedule new transfers for ~20 minutes.
    • A few SL5 machines also went down. No impact on service but a useful reminder to get them decommissioned properly.
    • It should be worth noting that two Echo storage nodes went down with no impact on service. This happened because their power cables weren’t correctly fitted.
    • GOCDB service was down for 45 minutes. It wasn’t directly a result of the power testing it was failed network switch (GOCDB is not on Tier-1 network, so this failure did not impact the Tier-1). The read only version (in Germany) stayed up. EGI broadcast was sent out a few minutes after the service went down.
Current operational status and issues
  • None.
Resolved Castor Disk Server Issues
  • None
Ongoing Castor Disk Server Issues
  • None.
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
  • See "Declared in the GOC DB"
Declared in the GOC DB
  • None
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Internal
    • DNS servers will be rolled out within the Tier1 network.

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
136408 cms in progress urgent 27/07/2018 27/07/2018 CMS_Data Transfers missing files at RAL WLCG
136366 mice in progress less urgent 25/07/2018 26/07/2018 Local Batch System Remove MICE Queue from RAL T1 Batch EGI
136199 lhcb in progress very urgent 18/07/2018 27/07/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
136028 cms in progress top priority 10/07/2018 27/07/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG
134685 dteam waiting for reply less urgent 23/04/2018 30/07/2018 Middleware please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 EGI
124876 ops in progress less urgent 07/11/2016 23/07/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
136427 atlas solved urgent 28/07/2018 28/07/2018 File Transfer UK RAL-LCG2: Transfer errors as destination WLCG
136359 none solved top priority 25/07/2018 25/07/2018 Other This TEST ALARM has been raised for testing GGUS alarm work flow after a new GGUS release. WLCG
136358 cms solved urgent 25/07/2018 27/07/2018 CMS_Facilities T1_UK_RAL WN-xrootd-access failure WLCG
136270 cms solved urgent 22/07/2018 23/07/2018 CMS_Facilities T1_UK_RAL xrootd access failures WLCG
136138 t2k.org solved urgent 16/07/2018 28/07/2018 File Access Extremely long download times for T2K files on tape at RALL - Part 2 EGI
136104 ops verified less urgent 13/07/2018 23/07/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
136002 cms closed urgent 09/07/2018 23/07/2018 CMS_Facilities T1_UK_RAL SE Xrootd read failure WLCG

Availability Report

Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-07-23 100 100 100 100 100 100
2018-07-24 100 100 97 100 100 100
2018-07-25 98 100 93 100 100 100
2018-07-26 100 100 100 100 100 100
2018-07-27 100 100 99 100 100 100
2018-07-28 100 100 96 100 84 100
2018-07-29 100 100 98 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-07-24 95 99
2018-07-25 98 100
2018-07-26 98 99
2018-07-27 98 99
2018-07-28 98 98
2018-07-29 100 98
2018-07-30 - -

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.