Tier1 Operations Report 2019-02-25

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 25th February 2019

Review of Issues during the week 18th February 2019 to the 25th February 2019.
  • Castor disk server were physically moved to make room for new procurements. This was done in a rolling manner during Tuesday 19th February. The data on a disk server was unavailable while it was being moved, but relatively little impact on LHCb was observed.
  • We have had two Castor disk server crashes since the move gdss776 and gdss783 both lhcbDst disk servers.
  • All but one of the ARC CEs has been upgraded. We are observing significantly less load on the machines, which we believe was the cause of most of the other issues observed.
  • CPU efficiency is improving. There is an ongoing discussion with CMS regarding some of their jobs. Their most problematic jobs involve large input files (> 10GB) which have metadata spread across the file (this is being worked on by CMS).
Current operational status and issues
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss811 LHCb lhcbDst d1t0 -
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss776 LHCb lhcbDst d1t0 -
Limits on concurrent batch system jobs.
  • ALICE - 1000
Notable Changes made since the last meeting.
  • NTR
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
ARC-CE 26729 Yes Outage 12/2/2019 16:30 19/2/2019 14:00 7 Days ARC-CE draining and upgrade to ARC v5.4.3
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - -
  • No ongoing downtime
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
139858 cms waiting for reply urgent 23/02/2019 25/02/2019 CMS_Data Transfers Failing transfers of 7 files from T1_UK_RAL_Disk to T2_FR_GRIF_LLR WLCG
139723 atlas in progress less urgent 15/02/2019 22/02/2019 Data Management - generic permissions on scratchdisk EGI
139672 other in progress less urgent 13/02/2019 20/02/2019 Middleware No LIGO pilots running at RAL EGI
139639 cms in progress very urgent 12/02/2019 21/02/2019 CMS_AAA WAN Access file open error at RAL WLCG
139476 mice in progress less urgent 01/02/2019 20/02/2019 Other LFC dump EGI
139306 dteam in progress less urgent 24/01/2019 22/02/2019 Monitoring perfsonar hosts need updating EGI
138665 mice on hold urgent 04/12/2018 30/01/2019 Middleware Problem accessing LFC at RAL EGI
138500 cms in progress urgent 26/11/2018 12/02/2019 CMS_Data Transfers Transfers failing from T2_PL_Swierk to RAL WLCG
138361 t2k.org in progress less urgent 19/11/2018 31/01/2019 Other RAL-LCG2: t2k.org LFC to DFC transition EGI
138033 atlas in progress urgent 01/11/2018 31/01/2019 Other singularity jobs failing at RAL EGI
137897 enmr.eu on hold urgent 23/10/2018 31/01/2019 Workload Management enmr.eu accounting at RAL EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
139793 ops solved less urgent 19/02/2019 21/02/2019 Operations [Rod Dashboard] Issue detected : org.sam.SRM-Put-ops@srm-alice.gridpp.rl.ac.uk EGI
139779 atlas verified less urgent 19/02/2019 21/02/2019 File Access File may be lost at RAL-LCG2-ECHO_DATADISK EGI
139538 cms closed urgent 05/02/2019 19/02/2019 CMS_Data Transfers Some transfers failing to RAL - SRM_AUTHORIZATION_FAILURE WLCG
139375 atlas closed urgent 29/01/2019 18/02/2019 Other RAL-LCG2 transfers fail with "the server responded with an error 500" WLCG
139245 cms closed urgent 21/01/2019 18/02/2019 CMS_Data Transfers Transfers failing from CNAF_Disk to RAL_Buffer WLCG

Availability Report

Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2019-02-19 100 100 100 100 90 100
2019-02-20 100 100 100 100 100 100
2019-02-21 100 100 100 100 100 100
2019-02-22 100 100 100 100 100 100
2019-02-23 100 100 100 100 100 100
2019-02-24 100 100 100 100 100 100
2019-02-25 58 58 58 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2019-01-23 100 98
2019-01-24 100 98
2019-01-25 100 98
2019-01-26 100 91
2019-01-27 100 97
2019-01-28 100 93
2019-01-29 100 98

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.