Tier1 Operations Report 2014-12-03

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 3rd December 2014

Review of Issues during the week 26th November to 3rd December 2014.
  • There were continuing problems with the CMS transfer manager machine (lcgclsf02) failing over the weekend (29th and 30th November) and again on Tuesday morning. These have been traced to the xrootd redirector process running on the box. We have moved the xrootd redirector to another machine.
  • CMS managed to fill up their Castor space allocation yesterday 2nd December. The VO was informed and they deleted some files.
Resolved Disk Server Issues
  • None
Current operational status and issues
  • Today CMS jobs have been overwhelming Castor with xrootd jobs. We hope to fix this by switching CMS to use RFIO and by reducing the number of CMS jobs running.
  • The problem of locking sessions in the Castor database that affected CMS & LHCb recurred on the morning of Tuesday 2nd December. Whilst this was transitory the cause has been understood and a fix will be provided in a future version of Castor.
Ongoing Disk Server Issues
  • None.
Notable Changes made this last week.
  • LHCb Castor headnodes were updated to SL6 on Tuesday 2nd December
  • Migration of CMS from 'B' to 'T10KD' tapes was completed.
Declared in the GOC DB
  • None.
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • The rollout of the RIP protocol to the Tier1 routers still has to be completed.
  • First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.(Provisional dates: week 12-16 January).
  • Castor headnode upgrades to SL6: (Assume 4 hour outage of Castor instance in each case for stager updates).
    • Tuesday 2nd Dec - LHCb; Tues 9th Dec - CMS; Wed 10th Dec - Atlas; Wednesday 7th Jan - GEN; Thursday 8th Jan - Nameserver (transparent - at risk)

Listing by category:

  • Databases:
    • A new database (Oracle RAC) has been set-up to host the Atlas 3D database. This is updated from CERN via Oracle GoldenGate. This system is yet to be brought into use. (Currently Atlas 3D/Frontier still uses the OGMA datase system, although this was also changed to update from CERN using Oracle Golden Gate.)
    • Switch LFC/3D to new Database Infrastructure.
    • Update to Oracle 11.2.0.4
  • Castor:
    • Update Castor headnodes to SL6.
    • Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
    • Update Castor to 2.1-14-latest, this depends on SL6 being deployed.
  • Networking:
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
    • Enable the RIP protocol for updating routing tables on the Tier1 routers.
  • Fabric
    • Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes finished; migration of GEN from 'A' to 'D' tapes to follow.)
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
    • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 26th November and 3rd December 2014.
Service Scheduled? Outage/At Risk Start End Duration Reason
srm-lhcb-tape.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, SCHEDULED OUTAGE 02/12/2014 10:00 02/12/2014 14:00 4 hours OS upgrade (SL6) on headnodes for LHCb Castor instance.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
110397 Green Less Urgent In Progress 2014-11-26 2014-11-27 Unable to access LFC webdav interface via browser dteam
110382 Green Less Urgent In Progress 2014-11-26 2014-11-26 RAL-LCG2: please reinstall your perfsonar hosts(s) N/A
109712 Amber Urgent In Progress 2014-10-29 2014-11-27 CMS Glexec exited with status 203; ...
109694 Green Urgent On hold 2014-11-03 2014-11-26 SNO+ gfal-copy failing for files at RAL
108944 Amber Urgent In Progress 2014-10-01 2014-11-26 CMS AAA access test failing at T1_UK_RAL
107935 Red Less Urgent On Hold 2014-08-27 2014-11-03 Atlas BDII vs SRM inconsistent storage capacity numbers
106324 Red Urgent On Hold 2014-06-18 2014-11-27 CMS pilots losing network connections at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
26/11/14 100 100 100 100 100 99 n/a
27/11/14 100 100 100 100 100 98 n/a
28/11/14 100 100 98.1 100 100 100 n/a
29/11/14 100 100 100 98.5 100 100 n/a Start of problems with CMS Castor scheduler headnode.
30/11/14 100 100 100 73.0 100 100 n/a Problems with CMS Castor scheduler headnode.
01/12/14 100 100 99.2 95.9 91.9 100 n/a CMS and LHCb had problems with DB locking issues.
02/12/14 100 100 100 88.68 78.32 100 100 LHCb headnodes updated to SL6, CMS full disk problems