Difference between revisions of "Tier1 Operations Report 2015-02-25"

Latest revision as of 14:07, 25 February 2015

RAL Tier1 Operations Report for 25th February 2015

Review of Issues during the week 18th to 25th February 2015.

A test of the problematic Tier1 router was carried out on Thursday morning, 19th Feb.
Yesterday (Tuesday) there was an outage of part of Castor as some racks containing disk servers (the 2011 batches) were shutdown while they were moved within the machine room to make room for new deliveries.
There were a couple of breaks in network connectivity between 07:00 and 08:00 yesterday (Tuesday 24th Feb) while core site network switches were upgraded.
On Wednesday (18th Feb) a single file was reportd lost to CMS. This file had been picked up by the Castor checksum checker.

Resolved Disk Server Issues

GDSS757 (CMSDisk - D1T0) failed early on Sunday morning (22nd Feb). Following checking it was returned to service the following morning.

Current operational status and issues

We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.

Ongoing Disk Server Issues

None

Notable Changes made this last week.

The 2011 disk servers were moved within the machine room to make space for new deliveries. As part of this the connections to these servers was migrated to the Tier1 mesh network.
On Wednesday (18th Feb) the Argus policy was updated to support the MICE reco role.
A Castor namesever box has been set-up to enable queries against Castor metadata to be made without affecting the throughput of production work.
A system has been set-up to provide Atlas with Castor information that is not supplied by the SRM.

Declared in the GOC DB

None

Advanced warning for other interventions

The following items are being discussed and are still to be formally scheduled and announced.

Investigate problems on the primary Tier1 router. Discussions with the vendor are ongoing.

Listing by category:

Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4
Castor:
- Update SRMs to new version (includes updating to SL6).
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Update Castor to 2.1-14-latest.
Networking:
- Resolve problems with primary Tier1 Router
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)

Entries in GOC DB starting since the last report.

Service	Scheduled?	Outage/At Risk	Start	End	Duration	Reason
Many Castor services (srm-alice, srm-atlas, srm-biomed, srm-dteam, srm-ilc, srm-lhcb-tape, srm-lhcb, srm-mice, srm-na62, srm-pheno, srm-snoplus, srm-superb, srm-t2k)	SCHEDULED	OUTAGE	24/02/2015 08:30	24/02/2015 17:06	8 hours and 36 minutes	Outage of some parts of the Castor storage while disk servers are moved to make space for new equipment.
Whole site	SCHEDULED	WARNING	19/02/2015 07:45	19/02/2015 08:30	45 minutes	Warning during test of network router. We expect a 5 minute break in connectivity to site sometime during this window.

Open GGUS Tickets (Snapshot during morning of meeting)


GGUS ID	Level	Urgency	State	Creation	Last Update	VO	Subject
111856	Green	Less Urgent	In Progress	2015-02-19	2015-02-24	LHCb	Stalled LHCb jobs over night
111699	Green	Less Urgent	In Progress	2015-02-10	2015-02-18	Atlas	gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
109694	Red	Urgent	Waiting Reply	2014-11-03	2015-02-24	SNO+	gfal-copy failing for files at RAL
108944	Red	Urgent	In Progress	2014-10-01	2015-02-24	CMS	AAA access test failing at T1_UK_RAL

Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud


Day	OPS	Alice	Atlas	CMS	LHCb	Atlas HC	CMS HC	Comment
18/02/15	100	100	100	100	100	100	100
19/02/15	100	100	98.0	100	100	98	97	Single SRM test failure
20/02/15	100	100	98.0	100	100	100	100	Single SRM test failure
21/02/15	100	100	100	100	100	100	99
22/02/15	100	100	98.0	100	100	100	99	Single SRM test failure
23/02/15	100	100	100	100	100	97	99
24/02/15	100	100	62.0	58.0	59.0	94	92	Planned move of disk server racks.

@@ Line 4: / Line 4: @@
 <!-- ************************************************************* ----->
-<!-- ***********Start Review of Issues during last three weeks*********** ----->
+<!-- ***********Start Review of Issues during last week*********** ----->
 {| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;"
 |-
 | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 18th to 25th February 2015.
 |}
-* On Monday morning the Tier1 data link failed. This is the link between the Tier1 routers and the UKLight router). All data transfers stopped. The problem lasted for around an hour from 09:00.
+* A test of the problematic Tier1 router was carried out on Thursday morning, 19th Feb.
-* A single file loss was reported to Atlas. This was picked up by the regular checksum valdiation process.
+* Yesterday (Tuesday) there was an outage of part of Castor as some racks containing disk servers (the 2011 batches) were shutdown while they were moved within the machine room to make room for new deliveries.
+* There were a couple of breaks in network connectivity between 07:00 and 08:00 yesterday (Tuesday 24th Feb) while core site network switches were upgraded.
+* On Wednesday (18th Feb) a single file was reportd lost to CMS. This file had been picked up by the Castor checksum checker.
 <!-- ***********End Review of Issues during last week*********** ----->
 <!-- *********************************************************** ----->
@@ Line 21: / Line 23: @@
 | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 |}
-* GDSS656 (LHCbRawRdst - D1T0) failed on Thursday afternoon (12th Feb). It was returned to service on the Saturday morning (14th Feb). A disk drive failed while verifying the RAID array.
+* GDSS757 (CMSDisk - D1T0) failed early on Sunday morning (22nd Feb). Following checking it was returned to service the following morning.
 <!-- ***********End Resolved Disk Server Issues*********** ----->
 <!-- ***************************************************** ----->
@@ Line 54: / Line 56: @@
 | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 |}
-* Condor update to version 8.2.7 applied to CEs and other internal Condor systems.
+* The 2011 disk servers were moved within the machine room to make space for new deliveries. As part of this the connections to these servers was migrated to the Tier1 mesh network.
-* Test/first roll-out deployments of condor 8.2.7 and cvmfs-client 2.1.20 on Worker nodes.
+* On Wednesday (18th Feb) the Argus policy was updated to support the MICE reco role.
-* Both production Atlas Frontier machines are now using Cronos as the Atlas 3D database backend instead of Ogma.
+* A Castor namesever box has been set-up to enable queries against Castor metadata to be made without affecting the throughput of production work.
+* A system has been set-up to provide Atlas with Castor information that is not supplied by the SRM.
 <!-- *************End Notable Changes made this last week************** ----->
 <!-- ****************************************************************** ----->
@@ Line 67: / Line 70: @@
 | style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 |}
-{| border=1 align=center
+None
-|- bgcolor="#7c8aaf"
-! Service
-! Scheduled?
-! Outage/At Risk
-! Start
-! End
-! Duration
-! Reason
-|-
-| Whole site
-| SCHEDULED
-| WARNING
-| 19/02/2015 07:45
-| 19/02/2015 08:30
-| 45 minutes
-| Warning during test of network router. We expect a 5 minute break in connectivity to site sometime during this window.
-|}
 <!-- **********************End GOC DB Entries************************** ----->
 <!-- ****************************************************************** ----->
@@ Line 98: / Line 84: @@
 |}
 <!-- ******* still to be formally scheduled and/or announced ******* ----->
-* Investigate problems on the primary Tier1 router. Discussions with the vendor are ongoing and a test is scheduled for tomorrow morning (19th Feb).
+* Investigate problems on the primary Tier1 router. Discussions with the vendor are ongoing.
-* Physical move of 2011 disk server racks to make space for new delivery.
 '''Listing by category:'''
 * Databases:
@@ Line 110: / Line 95: @@
 * Networking:
 ** Resolve problems with primary Tier1 Router
-** Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
 ** Make routing changes to allow the removal of the UKLight Router.
 ** Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
 * Fabric
-** Physical move of 2011 disk server racks to make space for new delivery.
 ** Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
 <!-- ***************End Advanced warning for other interventions*************** ----->
@@ Line 126: / Line 109: @@
 | style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report.
 |}
 {| border=1 align=center
 |- bgcolor="#7c8aaf"
@@ Line 138: / Line 119: @@
 ! Reason
 |-
-| Many  Castor services (srm-alice, srm-atlas, srm-biomed, srm-dteam, srm-ilc, srm-lhcb-tape, srm-lhcb, srm-mice, srm-na62, srm-pheno, srm-snoplus, srm-superb, srm-t2k
+| Many  Castor services (srm-alice, srm-atlas, srm-biomed, srm-dteam, srm-ilc, srm-lhcb-tape, srm-lhcb, srm-mice, srm-na62, srm-pheno, srm-snoplus, srm-superb, srm-t2k)
 | SCHEDULED
 | OUTAGE
@@ Line 168: / Line 149: @@
 |-style="background:#b7f1ce"
 ! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
+|-
+| 111856
+| Green
+| Less Urgent
+| In Progress
+| 2015-02-19
+| 2015-02-24
+| LHCb
+| Stalled LHCb jobs over night
 |-
 | 111699
 | Green
 | Less Urgent
-| Waiting Reply
+| In Progress
 | 2015-02-10
 | 2015-02-18
 | Atlas
 | gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
-|-
-| 111120
-| Green
-| Less Urgent
-| Waiting Reply
-| 2015-01-12
-| 2015-01-22
-| Atlas
-| large transfer errors from RAL-LCG2 to BNL-OSG2
 |-
 | 109694
@@ Line 192: / Line 173: @@
 | Waiting Reply
 | 2014-11-03
-| 2015-02-18
+| 2015-02-24
 | SNO+
 | gfal-copy failing for files at RAL
@@ Line 201: / Line 182: @@
 | In Progress
 | 2014-10-01
-| 2015-02-18
+| 2015-02-24
 | CMS
 | AAA access test failing at T1_UK_RAL

Difference between revisions of "Tier1 Operations Report 2015-02-25"

Latest revision as of 14:07, 25 February 2015

RAL Tier1 Operations Report for 25th February 2015

Navigation menu

Personal tools

Namespaces

Variants

Views

Actions

Search

Main GridPP website

Navigation

Tools