RAL Tier1 weekly operations castor 23/10/2015

From GridPP Wiki
Jump to: navigation, search

Operations News

  • 5 fresh nodes have been added to atlasTape
  • Attempted fix for LHCb's intermittent stage request failures - increase 'Stage Request' thread count from 4 to 12.

Operations Problems

  • More SRM DB duplicates continue. RA is developing an automatic script to clean up the usual case of these.
  • stager_qry has been running slowly for ATLAS
  • Multiple disk servers failures, mostly in the recently-deployed set of CV11 nodes deployed into atlasTape.
  • LHCb ticket re. low level of failures likely to be fail in nsmkdir. Look for timeouts in name server logs to prove where timeout is. Mitigation may be to increase thread count on NS. Be a little careful with this one as vastly increasing this could exceed connection limit to DB (number of NS daemons is quite high)

Blocking Issues

Planned, Scheduled and Cancelled Interventions

  • Tape backed nodes to SL6 - scheduled for Monday/Tuesday.
  • CASTOR 2.1.15
    • Currently scheduled for late this year/early next.
  • Upgrade of Oracle clients to 11.2.0.4

Issues to bring up at CASTOR F2F

  • Oracle 12
  • Slow running of stager commands when ATLAS under high load.

Advanced Planning

Tasks

  • CASTOR Face-to-face 2nd-3rd November.
  • CASTOR 2.1.15 implementation and testing

Interventions

Staffing

  • Castor on Call person next week
    • SdW
  • Staff absence/out of the office:
    • BD Monday, Thursday, Friday

New Actions

Existing Actions

  • SdW to revisit schema changes to improve SRM DB dup position W/C 12th Oct.
  • Production team check srm03 regarding lack of crl updates
  • RA to schedule no-op change to be included in quattor / persistent (CMSdisk)
  • BC to doc procedure for HW return to CASTOR
  • SdW to modify cleanlostfiles to log to syslog so we can track its use - under testing
  • RA/JJ to look at information provider re DiRAC (reporting disk only etc)
  • GS to arrange meeting castor/fab/production to discuss the decommissioning procedures
  • GS to investigate providing checks for /etc/noquattor on production nodes & checks for fetch-crl - ONGOING
  • BD to chase AD about using the space reporting thing we made for him
  • GS to investigate how/if we need to declare xrootd endpoints in GOCDB BDII

Completed actions

Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=RAL_Tier1_weekly_operations_castor_23/10/2015&oldid=9785"