Tier1 Operations Report 2018-01-24

RAL Tier1 Operations Report for 24th January 2018

Review of Issues during the week 18th January 2018 to 24th January 2018

• Patching for Spectre and Meltdown was taking place last week. (E.g. an outage of Castor for a few hours last Wednesday, 17th Jan).
• Problems since last week with the Atlas Castor instance. There has been a high rate of deletion requests, and the back end stager database has been struggling to keep up with the total load (reads, writes and deletes). Atlas have backed off at the moment. No more specific cause for the problems found but investigations ongoing.
• There will be an intervention on one of the BMS (Building Management Systems) in the R89 machine room on Wednesday. It has been intermittently faulty causing some system restarts. In one case the chillers failed to restart. This system manages the pumps. The intervention should take less than 30 minutes. On Tuesday one of the pumps will be reconnected ‘stand-alone’ and checked it works OK – and this will be used to maintain flow during the BMS controller swap.

Current operational status and issues

• Ongoing security patching.

Resolved Castor Disk Server Issues

• gdss736 (LHCbDst - D1T0) – rebuilt and back in production (RO)
• gdss728 (CMSDisk - D1T0) – rebuilt and back in production (RW)

Ongoing Castor Disk Server Issues

• gdss717 (CMSTape - D0T1) – multiple drive failure

Limits on concurrent batch system jobs.

• CMS Multicore 550

Notable Changes made since the last meeting.

• Security patching done or underway.

Entries in GOC DB starting since the last report.

• None

Declared in the GOC DB

Service	Scheduled?	Outage/At Risk	Start	End	Duration	Reason
srm-alice.gridpp.rl.ac.uk, srm-atlas.gridpp.rl.ac.uk, srm-biomed.gridpp.rl.ac.uk, srm-cert.gridpp.rl.ac.uk, srm-cms-disk.gridpp.rl.ac.uk, srm-cms.gridpp.rl.ac.uk, srm-dteam.gridpp.rl.ac.uk, srm-ilc.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-mice.gridpp.rl.ac.uk, srm-minos.gridpp.rl.ac.uk, srm-na62.gridpp.rl.ac.uk, srm-pheno.gridpp.rl.ac.uk, srm-preprod.gridpp.rl.ac.uk, srm-snoplus.gridpp.rl.ac.uk, srm-solid.gridpp.rl.ac.uk, srm-t2k.gridpp.rl.ac.uk,	SCHEDULED	OUTAGE	17/01/2018 10:00	17/01/2018 13:00	3 hours	Outage of Castor Storage to apply Security patches.
lcglb01.gridpp.rl.ac.uk, lcglb02.gridpp.rl.ac.uk, lcgwms04.gridpp.rl.ac.uk, lcgwms05.gridpp.rl.ac.uk,	SCHEDULED	OUTAGE	12/01/2018 10:00	19/01/2018 12:00	7 days, 2 hours	WMS Decommissioning RAL Tier1

Advanced warning for other interventions

The following items are being discussed and are still to be formally scheduled and announced.

Ongoing or Pending - but not yet formally announced:

Listing by category:

• Castor:
  • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
  • Move to generic Castor headnodes.
• Echo:
  • Update to next CEPH version ("Luminous").
• Networking
  • Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
• Services
  • Ending of WMS service (ongoing).
• Internal
  • DNS servers will be rolled out within the Tier1 network.
  • Infrastructure
• Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.

Open GGUS Tickets (Snapshot during morning of meeting)

Ticket-ID	Type	VO	Notified Site	Resp. Unit	Status	Priority	Creation	Last Update	ToI	Subject
132844	TEAM	atlas	RAL-LCG2	NGI_UK assign to:lcg-support@gridpp.rl.ac.uk	waiting for reply	urgent	2018-01-14 02:10:00	2018-01-16 16:51:00	Storage Systems	UK RAL-LCG2 DATADISK transfer errors "DESTINATION OVERWRITE srm-ifce err:"
132830	USER	cms	RAL-LCG2	NGI_UK	in progress	very urgent	2018-01-12 11:11:00	2018-01-16 16:36:00	CMS_AAA WAN Access	Reading issues T1_UK_RAL
132802	USER	cms	RAL-LCG2	NGI_UK	in progress	urgent	2018-01-11 10:30:00	2018-01-15 11:26:00	CMS_AAA WAN Access	Low HC xrootd success rates at T1_UK_RAL
132712	USER	other	RAL-LCG2	NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:Kashif.Mohammad@PHYSICS.OX.AC.UK	in progress	less urgent	2018-01-04 16:22:00	2018-01-16 15:01:00	Other	support for the hyperk VO (RAL-LCG2)
132589	TEAM	lhcb	RAL-LCG2	NGI_UK	in progress	very urgent	2017-12-21 06:45:00	2018-01-10 11:07:00	Local Batch System	Killed pilots at RAL
131815	USER	t2k.org	RAL-LCG2	NGI_UK	waiting for reply	less urgent	2017-11-13 14:42:00	2018-01-17 09:24:00	Storage Systems	Extremely long download times for T2K files on tape at RAL
127597	USER	cms	RAL-LCG2	NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:sexton@fnal.gov	on hold	urgent	2017-04-07 10:34:00	2017-10-05 09:14:00	File Transfer	Check networking and xrootd RAL-CERN performance
124876	USER	ops	RAL-LCG2	NGI_UK assign to:lcg-support@gridpp.rl.ac.uk	on hold	less urgent	2016-11-07 12:06:00	2017-11-13 16:55:00	Operations	[Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
117683	USER	none	RAL-LCG2	NGI_UK assign to:lcg-support@gridpp.rl.ac.uk	on hold	less urgent	2015-11-18 11:36:00	2018-01-03 15:26:00	Information System	CASTOR at RAL not publishing GLUE 2

Availability Report

Day	OPS	Alice	Atlas	CMS	LHCb	Atlas Echo	Comment
10/01/18	100	100	100	100	100	100
11/01/18	100	100	100	100	98	100	gdss736 (LHCb) failed
12/01/18	100	100	100	100	100	100
13/01/18	100	100	100	100	100	100
14/01/18	100	96	53	100	100	100
15/01/18	100	100	74	100	100	100
16/01/18	100	100	92	100	100	100

Hammercloud Test Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_UCORE, Template 841); CMS HC = CMS HammerCloud

Day	Atlas HC	CMS HC	Comment
10/01/18	100	100
11/01/18	100	100
12/01/18	100	100
13/01/18	100	100
14/01/18	100	100
15/01/18	100	99
16/01/18	85	99

Notes from Meeting.

• None yet.