Latest revision as of 10:59, 24 September 2014

RAL Tier1 Operations Report for 24th September 2014

Review of Issues during the week 17th to 24th September 2014.

On Thursday morning (around 07:40) there was a crash of the Atlas SRM database. The database immediately restarted on a different RAC node, and has subsequently been put back. This is a repeat of a known and reported Oracle bug.
There was some very high load on a number of disk servers in AtlasDataDisk, particularly during the latter part of last week. Local measures to improve the problem didn't help. Atlas are aware of issues caused by a partiular workflow.
Yesterday, Tues 23rd Sep, the Atlas Frontier squids were switched to use a new database system ("Cronos"). However, during the evening problems were found with the Frontier squids accessing the Cronos database and it was switched back to the former database ("Ogma"). The Cronos Atlas 3D database updates from CERN via Oracle "Goldengate" which supersedes the Oracle "Streams" method used to update Ogma.

Resolved Disk Server Issues

GDSS763 (AtlasDataDisk - D1T0) failed in the early morning of Thursday 18th Sep. It was restarted and tested but no fault found. The system was returned to service early the following afternoon (19th Sep).
GDSS720 (AtlasDataDisk - D1T0) failed on Sunday afternoon, 21st Sep. It was restarted and tested but no fault found. The system was returned to service at the end of Monday afternoon (22nd Sep).

Current operational status and issues

We are investigating why the Atlas Frontier systems had problems using the new Cronos database.

Ongoing Disk Server Issues

None.

Notable Changes made this last week.

VO Pheno have been enabled to use Castor.
The production FTS3 instance upgraded to 3.2.27

Declared in the GOC DB

Service	Scheduled?	Outage/At Risk	Start	End	Duration	Reason
arc-ce02, arc-ce03, arc-ce04	SCHEDULED	WARNING	09/10/2014 10:00	09/10/2014 11:00	1 hour	Upgrade of ARC CEs to version 4.2.0
arc-ce01	SCHEDULED	WARNING	07/10/2014 10:00	07/10/2014 11:00	1 hour	Upgrade of ARC CE to version 4.2.0

Advanced warning for other interventions

The following items are being discussed and are still to be formally scheduled and announced.

The rollout of the RIP protocol to the Tier1 routers still has to be completed.
Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday

30th September. Listing by category:

Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC).
- A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
- Switch LFC/3D to new Database Infrastructure.
Castor:
- Update Castor headnodes to SL6.
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).

Entries in GOC DB starting between the 17th and 24th September 2014.

Service	Scheduled?	Outage/At Risk	Start	End	Duration	Reason
lcgfts3.gridpp.rl.ac.uk,	SCHEDULED	WARNING	23/09/2014 10:00	23/09/2014 12:00	2 hours	Service At Risk during upgrade to FTS3 version 3.2.27.

Open GGUS Tickets (Snapshot during morning of meeting)


GGUS ID	Level	Urgency	State	Creation	Last Update	VO	Subject
108546	Green	Less Urgent	In Progress	2014-09-16	2014-09-22	Atlas	RAL-LCG2_HIMEM_SL6: production jobs failed
107935	Amber	Less Urgent	In Progress	2014-08-27	2014-09-02	Atlas	BDII vs SRM inconsistent storage capacity numbers
107880	Red	Less Urgent	In Progress	2014-08-26	2014-09-02	SNO+	srmcp failure
106324	Red	Urgent	In Progress	2014-06-18	2014-09-23	CMS	pilots losing network connections at T1_UK_RAL

Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud


Day	OPS	Alice	Atlas	CMS	LHCb	Atlas HC	CMS HC
17/09/14	100	100	100	100	100	98	96
18/09/14	100	100	100	100	100	99	99
19/09/14	100	100	100	100	100	100	98
20/09/14	100	100	100	100	100	96	97
21/09/14	100	100	100	100	100	97	97
22/09/14	100	100	100	100	100	95	98
23/09/14	100	100	100	100	100	98	98

@@ Line 9: / Line 9: @@
 | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 17th to 24th September 2014.
 |}
-* On Saturday (13th Sep) there was a problem with the Atlas Castor instance that persisted into the beginning of Sunday. A number of measures were taken to improve it, although the root cause remains unknown.
+* On Thursday morning (around 07:40) there was a crash of the Atlas SRM database. The database immediately restarted on a different RAC node, and has subsequently been put back. This is a repeat of a known and reported Oracle bug.
-* For the second half of last week there were problems with cream-ce02.
+* There was some very high load on a number of disk servers in AtlasDataDisk, particularly during the latter part of last week. Local measures to improve the problem didn't help. Atlas are aware of issues caused by a partiular workflow.
-* This morning (Wednesday 17th Sep) there was a problem with some machines that run as VMs - the symptom was that their networking stopped. Restarting the network fixed the problem. This is similar to a problem seen on the 30th August. The configuration of the network interface on these systems has been changed to workaround this. One of the systems affected was the argus server and this caused a problem for batch job submissions for an hour or so.
+* Yesterday, Tues 23rd Sep, the Atlas Frontier squids were switched to use a new database system ("Cronos"). However, during the evening problems were found with the Frontier squids accessing the Cronos database and it was switched back to the former database ("Ogma"). The Cronos Atlas 3D database updates from CERN via Oracle "Goldengate" which supersedes the Oracle "Streams" method used to update Ogma.
 <!-- ***********End Review of Issues during last week*********** ----->
 <!-- *********************************************************** ----->
@@ Line 22: / Line 22: @@
 | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 |}
-* None
+* GDSS763 (AtlasDataDisk - D1T0) failed in the early morning of Thursday 18th Sep. It was restarted and tested but no fault found. The system was returned to service early the following afternoon (19th Sep).
+* GDSS720 (AtlasDataDisk - D1T0) failed on Sunday afternoon, 21st Sep. It was restarted and tested but no fault found. The system was returned to service at the end of Monday afternoon (22nd Sep).
 <!-- ***********End Resolved Disk Server Issues*********** ----->
 <!-- ***************************************************** ----->
@@ Line 33: / Line 34: @@
 | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 |}
-* None.
+* We are investigating why the Atlas Frontier systems had problems using the new Cronos database.
 <!-- ***********End Current operational status and issues*********** ----->
 <!-- *************************************************************** ----->
@@ Line 55: / Line 56: @@
 | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 |}
-* VO Londongrid enabled on LFC.
+* VO Pheno have been enabled to use Castor.
+* The production FTS3 instance upgraded to 3.2.27
 <!-- *************End Notable Changes made this last week************** ----->
 <!-- ****************************************************************** ----->
@@ Line 66: / Line 68: @@
 | style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 |}
-* None
+{| border=1 align=center
+|- bgcolor="#7c8aaf"
+! Service
+! Scheduled?
+! Outage/At Risk
+! Start
+! End
+! Duration
+! Reason
+|-
+| arc-ce02, arc-ce03, arc-ce04
+| SCHEDULED
+| WARNING
+| 09/10/2014 10:00
+| 09/10/2014 11:00
+| 1 hour
+| Upgrade of ARC CEs to version 4.2.0
+|-
+| arc-ce01
+| SCHEDULED
+| WARNING
+| 07/10/2014 10:00
+| 07/10/2014 11:00
+| 1 hour
+| Upgrade of ARC CE to version 4.2.0
+|}
 <!-- **********************End GOC DB Entries************************** ----->
 <!-- ****************************************************************** ----->
@@ Line 83: / Line 112: @@
 * Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday
 th September.
-* The Atlas Frontier service will be switched to use the new database system that updates from CERN using Oracle "GoldenGate" on 24th Sep.
 '''Listing by category:'''
 * Databases:
@@ Line 110: / Line 138: @@
 | style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting between the 17th and 24th September 2014.
 |}
-* None
+{| border=1 align=center
+|- bgcolor="#7c8aaf"
+! Service
+! Scheduled?
+! Outage/At Risk
+! Start
+! End
+! Duration
+! Reason
+|-
+| lcgfts3.gridpp.rl.ac.uk,
+| SCHEDULED
+| WARNING
+| 23/09/2014 10:00
+| 23/09/2014 12:00
+| 2 hours
+| Service At Risk during upgrade to FTS3 version 3.2.27.
+|}
 <!-- **********************End GOC DB Entries************************** ----->
 <!-- ****************************************************************** ----->
@@ Line 131: / Line 176: @@
 | In Progress
 | 2014-09-16
-| 2014-09-16
+| 2014-09-22
 | Atlas
 | RAL-LCG2_HIMEM_SL6: production jobs failed
 |-
 | 107935
-| Yellow
+| Amber
 | Less Urgent
 | In Progress
@@ Line 145: / Line 190: @@
 |-
 | 107880
-| Amber
+| Red
 | Less Urgent
 | In Progress
@@ Line 156: / Line 201: @@
 | Red
 | Urgent
-| On Hold
+| In Progress
 | 2014-06-18
-| 2014-08-14
+| 2014-09-23
 | CMS
 | pilots losing network connections at T1_UK_RAL
-|-
-| 105405
-| Red
-| Urgent
-| On Hold
-| 2014-05-14
-| 2014-09-12
-|
-| Please check your Vidyo router firewall configuration
 |}
 <!-- **********************End GGUS Tickets************************** ----->
@@ Line 188: / Line 224: @@
 |-style="background:#b7f1ce"
 ! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment
-|-
-| 10/09/14 || 100 || 100 || style="background-color: lightgrey;" | 99.2 || 100 || 100 || 96 || 97 || Single SRM test failure on GET - [SRM_FILE_BUSY]
 |-
 | 17/09/14 || 100 || 100 || 100 || 100 || 100 || 98 || 96 ||

Difference between revisions of "Tier1 Operations Report 2014-09-24"

Latest revision as of 10:59, 24 September 2014

RAL Tier1 Operations Report for 24th September 2014

Navigation menu

Personal tools

Namespaces

Variants

Views

Actions

Search

Main GridPP website

Navigation

Tools