Tier1 Operations Report 2014-09-24

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 24th September 2014

Review of Issues during the week 17th to 24th September 2014.
  • On Thursday morning (around 07:40) there was a crash of the Atlas SRM database. The database immediately restarted on a different RAC node, and has subsequently been put back. This is a repeat of a known and reported Oracle bug.
  • There was some very high load on a number of disk servers in AtlasDataDisk, particularly during the latter part of last week. Local measures to improve the problem didn't help. Atlas are aware of issues caused by a partiular workflow.
  • Yesterday, Tues 23rd Sep, the Atlas Frontier squids were switched to use a new database system ("Cronos"). However, during the evening problems were found with the Frontier squids accessing the Cronos database and it was switched back to the former database ("Ogma"). The Cronos Atlas 3D database updates from CERN via Oracle "Goldengate" which supersedes the Oracle "Streams" method used to update Ogma.
Resolved Disk Server Issues
  • GDSS763 (AtlasDataDisk - D1T0) failed in the early morning of Thursday 18th Sep. It was restarted and tested but no fault found. The system was returned to service early the following afternoon (19th Sep).
  • GDSS720 (AtlasDataDisk - D1T0) failed on Sunday afternoon, 21st Sep. It was restarted and tested but no fault found. The system was returned to service at the end of Monday afternoon (22nd Sep).
Current operational status and issues
  • We are investigating why the Atlas Frontier systems had problems using the new Cronos database.
Ongoing Disk Server Issues
  • None.
Notable Changes made this last week.
  • VO Pheno have been enabled to use Castor.
  • The production FTS3 instance upgraded to 3.2.27
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
arc-ce02, arc-ce03, arc-ce04 SCHEDULED WARNING 09/10/2014 10:00 09/10/2014 11:00 1 hour Upgrade of ARC CEs to version 4.2.0
arc-ce01 SCHEDULED WARNING 07/10/2014 10:00 07/10/2014 11:00 1 hour Upgrade of ARC CE to version 4.2.0


Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • The rollout of the RIP protocol to the Tier1 routers still has to be completed.
  • Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday

30th September. Listing by category:

  • Databases:
    • Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC).
    • A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
    • Switch LFC/3D to new Database Infrastructure.
  • Castor:
    • Update Castor headnodes to SL6.
    • Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
  • Networking:
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
    • Enable the RIP protocol for updating routing tables on the Tier1 routers.
  • Fabric
    • Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
    • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 17th and 24th September 2014.
Service Scheduled? Outage/At Risk Start End Duration Reason
lcgfts3.gridpp.rl.ac.uk, SCHEDULED WARNING 23/09/2014 10:00 23/09/2014 12:00 2 hours Service At Risk during upgrade to FTS3 version 3.2.27.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
108546 Green Less Urgent In Progress 2014-09-16 2014-09-22 Atlas RAL-LCG2_HIMEM_SL6: production jobs failed
107935 Amber Less Urgent In Progress 2014-08-27 2014-09-02 Atlas BDII vs SRM inconsistent storage capacity numbers
107880 Red Less Urgent In Progress 2014-08-26 2014-09-02 SNO+ srmcp failure
106324 Red Urgent In Progress 2014-06-18 2014-09-23 CMS pilots losing network connections at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
17/09/14 100 100 100 100 100 98 96
18/09/14 100 100 100 100 100 99 99
19/09/14 100 100 100 100 100 100 98
20/09/14 100 100 100 100 100 96 97
21/09/14 100 100 100 100 100 97 97
22/09/14 100 100 100 100 100 95 98
23/09/14 100 100 100 100 100 98 98