RAL Tier1 weekly operations castor 26/01/2018

Draft agenda

1. Problems encountered this week

2. Upgrades/improvements made this week

3. What are we planning to do next week? 4. Long-term project updates (if not already covered)

  1. SL5 elimination from CIP and tape verification server
  2. CASTOR stress test improvement
  3. Generic CASTOR headnode setup
  4. Aquilonised headnodes

5. Special topics

6. Actions

7. Anything for CASTOR-Fabric?

8. AoTechnicalB

9. Availability for next week

10. On-Call

11. AoOtherB

Operation problems

gdss762 (atlasStripInput) - running RAID verify.

gdss736 (lhcbDst) - Back in production

fdsdss34, 35, 36 - Down following reboot for patching. fdsdss34, 35, abck in prod, 36 stll down

The ACSSS tape library control server (buxton1) failed and refused to reboot. This created a backlog of canbemigrs which was eventually processed after the server was back up and running

Atlas Stager DB load problems continued. GC was not moving and the whole DISKCOPY table had to be recreated as it was fragmented RT201623

_----------------------------

CMS still having trouble with remote xrootd access - two GGUS tickets.

ATLAS - large pileup in the DB - lots of queries (printmigrationstatus, printdiskcopy) running very slowly (> 1 day). This is thought to be due to DB tables growing unreasonably large. The findFailedMigrations script returns a great deal of stuff - is this related? - Fiddling with execution plans made things a very little bit better. RA and MLF to look at later.

Operation news

Patching for Meltdown/Spectre done on both Tier 1 and Facilities.

Still some cleanup to do for assorted admin boxes.

New CIP currently under construction, will be ready Real Soon Now.

Plans for next week

RA: Skiing.

GP: Fix-up on Aqulion SRM profiles.

GP: Roll-up of remaining unpatched CASTOR machines

--

Patching for Meltdown/Spectre

Long-term projects

CASTOR stress test improvement - Script writing, awaiting testing and validation

Headnode migration to Aquilon - Stager configuration almost complete, awaiting testing.

Target: Combined headnodes running on SL7/Aquilon - implement CERN-style 'Macro' headnodes.

Draining of remainder of 12 generation HW - waiting on CMS migration to Echo. No more draining ongoing.

Actions

RA to check if the disk server setting, changed to bring disk servers back more quickly after a CASTOR shutdown, is still in place

RA/BD: Run GFAL unit tests against CASTOR. Get them here: https://gitlab.cern.ch/dmc/gfal2/tree/develop/test/

GP/RA to chase up Fabric team about getting action on RT197296(use fdscspr05 as the preprod stager - replacement for ccse08)

Staffing

GP On call

RA out next week