Difference between revisions of "Tier1 Operations Report 2014-10-01"

From GridPP Wiki
Jump to: navigation, search
(→‎)
 
(19 intermediate revisions by one user not shown)
Line 1: Line 1:
−
=RAL Tier1 Operations Report for 1st Octomber 2014=
+
=RAL Tier1 Operations Report for 1st October 2014=
 
__NOTOC__
 
__NOTOC__
 
====== ======
 
====== ======
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 24th September to 1st October 2014.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 24th September to 1st October 2014.
 
|}
 
|}
−
* On Thursday morning (around 07:40) there was a crash of the Atlas SRM database. The database immediately restarted on a different RAC node, and has subsequently been put back. This is a repeat of a known and reported Oracle bug.
+
* There were problems with the Atlas Castor instance over the weekend which was linked to the draining of a disk server. This was followed by very heavy load experienced by that server.
−
* There was some very high load on a number of disk servers in AtlasDataDisk, particularly during the latter part of last week. Local measures to improve the problem didn't help. Atlas are aware of issues caused by a partiular workflow.
+
* There were problems with the SAM tests on the CEs from Tuesday through to Wednesday. Believed due to a problem in one of the site BDIIs.
−
* Yesterday, Tues 23rd Sep, the Atlas Frontier squids were switched to use a new database system ("Cronos"). However, during the evening problems were found with the Frontier squids accessing the Cronos database and it was switched back to the former database ("Ogma"). The Cronos Atlas 3D database updates from CERN via Oracle "Goldengate" which supersedes the Oracle "Streams" method used to update Ogma.
+
* This morning, Wed 1st October, one of the Neptune standby databases crashed. Cause unknown. A Service Request has been opened with Oracle.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 22: Line 22:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
−
* GDSS763 (AtlasDataDisk - D1T0) failed in the early morning of Thursday 18th Sep. It was restarted and tested but no fault found. The system was returned to service early the following afternoon (19th Sep).
+
* GDSS763 (AtlasDataDisk - D1T0) failed during Saturday evening. It was restarted and tested but no fault found. The system was returned to service yesterday (30th Sep).
−
* GDSS720 (AtlasDataDisk - D1T0) failed on Sunday afternoon, 21st Sep. It was restarted and tested but no fault found. The system was returned to service at the end of Monday afternoon (22nd Sep).
+
* GDSS720 (AtlasDataDisk - D1T0) was reported as having had problems (and being returned to service) in last week's report. This server has suffered four failures in the last 14 months and is being drained pending further investigation.
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 56: Line 56:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
−
* VO Pheno have been enabled to use Castor.
+
* One batch of worker nodes (64 machines) have had Linux cgroups configured to enforce memory limits. (Before it was just advisory).
−
* The production FTS3 instance upgraded to 3.2.27
+
* Oracle patches (PSU) applied to the standby Neptune database (Castor Atlas & GEN) yesterday (Tuesday 30th Sep).
 +
* Assorted system (Linux) updates.
 +
* Migration of CMS data from 'B' to 'D' tapes is continuing.
 +
* There was a successful UPS/Generator load test this morning (Wed 1st Oct).
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 110: Line 113:
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
* The rollout of the RIP protocol to the Tier1 routers still has to be completed.  
 
* The rollout of the RIP protocol to the Tier1 routers still has to be completed.  
−
* Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday  
+
* Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. This was proposed for yesterday (Tuesday 30th September) but has been delayed.
−
30th September.  
+
* First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.
 
'''Listing by category:'''
 
'''Listing by category:'''
 
* Databases:
 
* Databases:
−
** Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC).
+
** Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC). (Underway).
 
** A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
 
** A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
 
** Switch LFC/3D to new Database Infrastructure.
 
** Switch LFC/3D to new Database Infrastructure.
Line 171: Line 174:
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
|-
 
|-
−
| 108546
+
| 108944
 +
| Green
 +
| Urgent
 +
| In Progress
 +
| 2014-10-01
 +
| 2014-10-01
 +
| CMS
 +
| AAA access test failing at T1_UK_RAL
 +
|-
 +
| 108886
 
| Green
 
| Green
 +
| Less Urgent
 +
| In Progress
 +
| 2014-09-29
 +
| 2014-09-30
 +
| dteam
 +
| RAL stratum one, cernvmfs.gridpp. connections
 +
|-
 +
| 108845
 +
| Green
 +
| Urgent
 +
| In Progress
 +
| 2014-09-27
 +
| 2014-09-30
 +
| Atlas
 +
| RAL-LCG2: Source connection timeout plus globus_ftp_client error
 +
|-
 +
| 108546
 +
| Yellow
 
| Less Urgent
 
| Less Urgent
 
| In Progress
 
| In Progress
Line 181: Line 211:
 
|-
 
|-
 
| 107935
 
| 107935
−
| Amber
+
| Red
 
| Less Urgent
 
| Less Urgent
 
| In Progress
 
| In Progress
Line 237: Line 267:
 
| 29/09/14 || 100 || 100 || style="background-color: lightgrey;" | 97.9 || 100 || 100 || 97 || n/a || Atlas test failure.
 
| 29/09/14 || 100 || 100 || style="background-color: lightgrey;" | 97.9 || 100 || 100 || 97 || n/a || Atlas test failure.
 
|-
 
|-
−
| 30/09/14 || style="background-color: lightgrey;" | 94.6 || 100 || 100 || style="background-color: lightgrey;" | 95.5 || 100 || 98 || n/a || Multiple CE test failures.
+
| 30/09/14 || style="background-color: lightgrey;" | 94.6 || 100 || 100 || style="background-color: lightgrey;" | 95.5 || 100 || 98 || n/a || Multiple CE test failures. Possibly due to problem on one of our site BDIIs.
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->

Latest revision as of 10:06, 8 October 2014

RAL Tier1 Operations Report for 1st October 2014

Review of Issues during the week 24th September to 1st October 2014.
  • There were problems with the Atlas Castor instance over the weekend which was linked to the draining of a disk server. This was followed by very heavy load experienced by that server.
  • There were problems with the SAM tests on the CEs from Tuesday through to Wednesday. Believed due to a problem in one of the site BDIIs.
  • This morning, Wed 1st October, one of the Neptune standby databases crashed. Cause unknown. A Service Request has been opened with Oracle.
Resolved Disk Server Issues
  • GDSS763 (AtlasDataDisk - D1T0) failed during Saturday evening. It was restarted and tested but no fault found. The system was returned to service yesterday (30th Sep).
  • GDSS720 (AtlasDataDisk - D1T0) was reported as having had problems (and being returned to service) in last week's report. This server has suffered four failures in the last 14 months and is being drained pending further investigation.
Current operational status and issues
  • We are investigating why the Atlas Frontier systems had problems using the new Cronos database.
Ongoing Disk Server Issues
  • None.
Notable Changes made this last week.
  • One batch of worker nodes (64 machines) have had Linux cgroups configured to enforce memory limits. (Before it was just advisory).
  • Oracle patches (PSU) applied to the standby Neptune database (Castor Atlas & GEN) yesterday (Tuesday 30th Sep).
  • Assorted system (Linux) updates.
  • Migration of CMS data from 'B' to 'D' tapes is continuing.
  • There was a successful UPS/Generator load test this morning (Wed 1st Oct).
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
arc-ce02, arc-ce03, arc-ce04 SCHEDULED WARNING 09/10/2014 10:00 09/10/2014 11:00 1 hour Upgrade of ARC CEs to version 4.2.0
arc-ce01 SCHEDULED WARNING 07/10/2014 10:00 07/10/2014 11:00 1 hour Upgrade of ARC CE to version 4.2.0


Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • The rollout of the RIP protocol to the Tier1 routers still has to be completed.
  • Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. This was proposed for yesterday (Tuesday 30th September) but has been delayed.
  • First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.

Listing by category:

  • Databases:
    • Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC). (Underway).
    • A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
    • Switch LFC/3D to new Database Infrastructure.
  • Castor:
    • Update Castor headnodes to SL6.
    • Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
  • Networking:
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
    • Enable the RIP protocol for updating routing tables on the Tier1 routers.
  • Fabric
    • Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
    • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 24th September and 1st October 2014.
Service Scheduled? Outage/At Risk Start End Duration Reason
Whole Site SCHEDULED WARNING 01/10/2014 10:00 01/10/2014 12:00 2 hours RAL Tier1 site in warning state due to UPS/generator test.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
108944 Green Urgent In Progress 2014-10-01 2014-10-01 CMS AAA access test failing at T1_UK_RAL
108886 Green Less Urgent In Progress 2014-09-29 2014-09-30 dteam RAL stratum one, cernvmfs.gridpp. connections
108845 Green Urgent In Progress 2014-09-27 2014-09-30 Atlas RAL-LCG2: Source connection timeout plus globus_ftp_client error
108546 Yellow Less Urgent In Progress 2014-09-16 2014-09-22 Atlas RAL-LCG2_HIMEM_SL6: production jobs failed
107935 Red Less Urgent In Progress 2014-08-27 2014-09-02 Atlas BDII vs SRM inconsistent storage capacity numbers
107880 Red Less Urgent In Progress 2014-08-26 2014-09-02 SNO+ srmcp failure
106324 Red Urgent In Progress 2014-06-18 2014-09-23 CMS pilots losing network connections at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
24/09/14 100 100 100 100 100 100 99
25/09/14 100 100 100 100 100 99 100
26/09/14 100 100 93.5 100 100 98 98 Problems with Atlas Castor instance (triggered by draining).
27/09/14 100 100 86.1 100 100 92 n/a Problems with Atlas Castor instance (triggered by draining).
28/09/14 100 100 85.9 100 100 93 n/a Problems with Atlas Castor instance (triggered by draining).
29/09/14 100 100 97.9 100 100 97 n/a Atlas test failure.
30/09/14 94.6 100 100 95.5 100 98 n/a Multiple CE test failures. Possibly due to problem on one of our site BDIIs.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2014-10-01&oldid=6201"