Difference between revisions of "Tier1 Operations Report 2016-06-01"

Latest revision as of 07:42, 7 June 2016

RAL Tier1 Operations Report for 1st June 2016

Review of Issues during the week 25th May to 1st June 2016.

Last week we reported a problem with the Tape Library control software that had started crashing. On Thursday (26th) we switched to using a backup server for the connection to the tape libraries. This still crashed but less frequently. Since then we have been running in this configuration. Crashes have still occurred but we have a re-starter in place plus monitoring so that we are aware of and can fix problems. Since then the tape libraries have delivered a reasonable service - although this is because we are managing the situation. This morning an engineer has come to fix a couple of hardware faults which are not show stoppers but need resolving. It is not clear if this will help the problem of the software crashing. We have put a 'warning' in the GOC DB for some hours today to cover this time. We continue to follow up with the vendor.

Resolved Disk Server Issues

GDSS698 (LHCbDst - D1T0) crashed overnight Monday/Tuesday 30/31 May. It was returned to service this morning following BIOS and firmware updates.
GDSS718 (LHCbDst - D1T0) crashed ysterday evening (Tuesday 31st May. It was returned to service this morning following BIOS and firmware updates.

Current operational status and issues

There is a problem seen by LHCb of a low but persistent rate of failure when copying the results of batch jobs to Castor. There is also a further problem that sometimes occurs when these (failed) writes are attempted to storage at other sites. A recent modification has improved, but not completed fixed this.
The intermittent, low-level, load-related packet loss seen over external connections is still being tracked. Likewise we have been working to understand some remaining low level of packet loss seen within a part of our Tier1 network.

Ongoing Disk Server Issues

None

Notable Changes made since the last meeting.

The migration of Atlas data from "C" to "D" tapes is ongoing. We have migrated around 400 of the 1300 tapes so far. This migration has carried on through this last week despite the tape library problems.
The HAProxy load balancers are now fully in use in front of the production FTS3 service.
New tape pools have been set-up for Atlas and LHCb.

Declared in the GOC DB

Service	Scheduled?	Outage/At Risk	Start	End	Duration	Reason
lcgwms06.gridpp.rl.ac.uk,	SCHEDULED	OUTAGE	01/06/2016 11:00	30/06/2016 11:00	29 days,	Server lcgwms06.gridpp.rl.ac.uk Decommissioning

Advanced warning for other interventions

The following items are being discussed and are still to be formally scheduled and announced.

Decommissioning of "GEN Scratch" storage in Castor. (Formally announced by EGI broadcast). Write access to this area has now been stopped in preparation for completely stopping access on the 20th June.
Decommissioning of lcgwms06. This is being stopped from receiving new work today (1st June).

Listing by category:

Databases:
- Switch LFC/3D to new Database Infrastructure.
Castor:
- Update SRMs to new version, including updating to SL6. This will be done after the Castor 2.1.15 update.
- Update to Castor version 2.1.15. This awaits successful resolution and testing of the new version.
- Migration of data from T10KC to T10KD tapes (Affects Atlas & LHCb data).
Networking:
- Replace the UKLight Router. Then upgrade the 'bypass' link to the RAL border routers to 2*40Gbit.
Fabric
- Firmware updates on remaining EMC disk arrays (Castor, LFC)
Grid Services
- Once the use of the Load Balancer (HAProxy) has been proven for the test FTS service it will be extended to other services.

Entries in GOC DB starting since the last report.

Service	Scheduled?	Outage/At Risk	Start	End	Duration	Reason
lcgwms06.gridpp.rl.ac.uk,	SCHEDULED	OUTAGE	01/06/2016 11:00	30/06/2016 11:00	29 days,	Server lcgwms06.gridpp.rl.ac.uk Decommissioning
All Castor endpoints (SRM) with tape.	SCHEDULED	WARNING	01/06/2016 10:00	01/06/2016 15:00	5 hours	Expected Break in Tape Access for Around Two Hours During this Time Window while Engineer Attends Tape Library.
All Castor endpoints (SRM) with tape.	UNSCHEDULED	WARNING	26/05/2016 18:00	27/05/2016 17:00	23 hours	Ongoing problems with tape library. Tape mounts may be delayed.
All Castor endpoints (SRM) with tape.	UNSCHEDULED	WARNING	25/05/2016 19:30	26/05/2016 14:00	18 hours and 30 minutes	At risk on tape system overnight following problem mounting tapes.

Open GGUS Tickets (Snapshot during morning of meeting)


GGUS ID	Level	Urgency	State	Creation	Last Update	VO	Subject
121687	Green	Less Urgent	On Hold	2016-05-20	2016-05-23		packet loss problems seen on RAL-LCG perfsonar
120810	Amber	Urgent	In Progress	2016-04-13	2016-05-24	Biomed	Decommissioning of SE srm-biomed.gridpp.rl.ac.uk - forbid write access for biomed users
120350	Green	Less Urgent	In Progress	2016-03-22	2016-05-06	LSST	Enable LSST at RAL
119841	Red	Less Urgent	On Hold	2016-03-01	2016-04-26	LHCb	HTTP support for lcgcadm04.gridpp.rl.ac.uk
117683	Yellow	Less Urgent	On Hold	2015-11-18	2016-04-05		CASTOR at RAL not publishing GLUE 2

Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 729); CMS HC = CMS HammerCloud


Day	OPS	Alice	Atlas	CMS	LHCb	Atlas HC	CMS HC
25/05/16	100	100	100	100	100	100	95
26/05/16	100	100	100	100	100	100	100
27/05/16	100	100	100	100	100	100	100
28/05/16	100	100	100	100	100	100	100
29/05/16	100	100	100	100	100	100	N/A
30/05/16	100	100	100	100	100	100	100
31/05/16	100	100	100	100	100	100	100

@@ Line 1: / Line 1: @@
-==RAL Tier1 Operations Report for 25th May 2016==
+==RAL Tier1 Operations Report for 1st June 2016==
 __NOTOC__
 ====== ======
@@ Line 9: / Line 9: @@
 | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 25th May to 1st June 2016.
 |}
-* On Friday (20th) Atlas put batch queues offline. This can be seen in the HammerCloud results below. Problem traced to Atlas Castor instance which was fixed during the day.
+* Last week we reported a problem with the Tape Library control software that had started crashing. On Thursday (26th) we switched to using a backup server for the connection to the tape libraries. This still crashed but less frequently. Since then we have been running in this configuration. Crashes have still occurred but we have a re-starter in place plus monitoring so that we are aware of and can fix problems. Since then the tape libraries have delivered a reasonable service - although this is because we are managing the situation. This morning an engineer has come to fix a couple of hardware faults which are not show stoppers but need resolving. It is not clear if this will help the problem of the software crashing. We have put a 'warning' in the GOC DB for some hours today to cover this time. We continue to follow up with the vendor.
-* Also on Friday (20th) there was a problem seen on a disk servers in CMSDisk that was running out of xroot connections. This limit was increased although that change was not picked up by all servers in the service class. There was a further problem with one disk server over the weekend which was resolved on Monday by forcing the recent change to be picked up by all (CMSDisk) disk servers.
-* Atlas put our batch queues offline yesterday (24th May). This was chased to an error in the new code that creates the 'machine job features'. This piece of code was disabled this morning and the problem fixed.
-* This morning (25th) we have a problem with the Tape Library control software that is crashing. Investigations are underway. Tape access is affected and a call has been opened with Oracle.
 <!-- ***********End Review of Issues during last week*********** ----->
 <!-- *********************************************************** ----->
@@ Line 23: / Line 20: @@
 | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 |}
-* GDSS635 (AtlasTape - D0T1) which had failed in the early hours of the 20th April was returned to service on Monday afternoon (23rd May). All its disks have been replaced and it has re-run the acceptance tests for a week. However, the next day, Tuesday (24th), the server was found not to be working correctly. It was taken out of production for a software problem to be identified and fixed. It was returned to production this morning.
+* GDSS698 (LHCbDst - D1T0) crashed overnight Monday/Tuesday 30/31 May. It was returned to service this morning following BIOS and firmware updates.
-* GDSS619 (GenTape - D0T1) failed on 26th April. As for GDSS635 (above) all its disks have been replaced and it has re-run the acceptance tests for a week. It was also returned to service on Monday (23rd May).
+* GDSS718 (LHCbDst - D1T0) crashed ysterday evening (Tuesday 31st May. It was returned to service this morning following BIOS and firmware updates.
-* GDSS664 (AtlasTape - D0T1) failed on Monday morning, 16th May. A verification of the RAID array found a faulty disk which has been replaced. It was returned to service last Wednesday (18th May).
-* GDSS727 (CMSDIsk reported 'FSProbe' problems on Thursday (19th May) and was taken out or service. Following the replacement of two disk drives it was returned to service on Sunday evening (22nd May).
 <!-- ***********End Resolved Disk Server Issues*********** ----->
 <!-- ***************************************************** ----->
@@ Line 49: / Line 44: @@
 | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues
 |}
-* None.
+* None
 <!-- ***************End Ongoing Disk Server Issues**************** ----->
 <!-- ************************************************************* ----->
@@ Line 60: / Line 55: @@
 | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 |}
-* The migration of Atlas data from "C" to "D" tapes is ongoing.
+* The migration of Atlas data from "C" to "D" tapes is ongoing. We have migrated around 400 of the 1300 tapes so far. This migration has carried on through this last week despite the tape library problems.
-* Write access to GenScratch has been stopped.
+* The HAProxy load balancers are now fully in use in front of the production FTS3 service.
-* One (of two) new batches of Worker Nodes has been put into production on Monday/Tuesday of this week.
+* New tape pools have been set-up for Atlas and LHCb.
 <!-- *************End Notable Changes made this last week************** ----->
 <!-- ****************************************************************** ----->
@@ Line 104: / Line 99: @@
 |}
 <!-- ******* still to be formally scheduled and/or announced ******* ----->
-* The Castor 2.1.15 update is pending. Testing has shown a database related performance issue which is being followed up. We await successful resolution of that problem and completion of testing before scheduling. Following advice from the developers we will not upgrade the SRMs before the Castor 2.1.15 upgrade.
 * Decommissioning of "GEN Scratch" storage in Castor. (Formally announced by EGI broadcast). Write access to this area has now been stopped in preparation for completely stopping access on the 20th June.
-* Decommissioning of lcgwms06. This will leave two WMS systems remaining in service.
+* Decommissioning of lcgwms06. This is being stopped from receiving new work today (1st June).
-* The HAProxy Load Balancer will be added in front of the Production FTS3 Service.
 '''Listing by category:'''
 * Databases:
 ** Switch LFC/3D to new Database Infrastructure.
 * Castor:
-** Update SRMs to new version (includes updating to SL6).
+** Update SRMs to new version, including updating to SL6. This will be done after the Castor 2.1.15 update.
-** Update to Castor version 2.1.15.
+** Update to Castor version 2.1.15. This awaits successful resolution and testing of the new version.
 ** Migration of data from T10KC to T10KD tapes (Affects Atlas & LHCb data).
 * Networking:
@@ Line 131: / Line 124: @@
 | style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report.
 |}
-* None
+{| border=1 align=center
+|- bgcolor="#7c8aaf"
+! Service
+! Scheduled?
+! Outage/At Risk
+! Start
+! End
+! Duration
+! Reason
+|-
+| lcgwms06.gridpp.rl.ac.uk,
+| SCHEDULED
+| OUTAGE
+| 01/06/2016 11:00
+| 30/06/2016 11:00
+| 29 days,
+| Server lcgwms06.gridpp.rl.ac.uk Decommissioning
+|-
+| All Castor endpoints (SRM) with tape.
+| SCHEDULED
+| WARNING
+| 01/06/2016 10:00
+| 01/06/2016 15:00
+| 5 hours
+| Expected Break in Tape Access for Around Two Hours During this Time Window while Engineer Attends Tape Library.
+|-
+| All Castor endpoints (SRM) with tape.
+| UNSCHEDULED
+| WARNING
+| 26/05/2016 18:00
+| 27/05/2016 17:00
+| 23 hours
+| Ongoing problems with tape library. Tape mounts may be delayed.
+|-
+| All Castor endpoints (SRM) with tape.
+| UNSCHEDULED
+| WARNING
+| 25/05/2016 19:30
+| 26/05/2016 14:00
+| 18 hours and 30 minutes
+| At risk on tape system overnight following problem mounting tapes.
+|}
 <!-- **********************End GOC DB Entries************************** ----->
 <!-- ****************************************************************** ----->
@@ Line 155: / Line 189: @@
 |
 | packet loss problems seen on RAL-LCG perfsonar
-|-
-| 120954
-| Green
-| Less Urgent
-| In Progress
-| 2016-04-21
-| 2016-05-24
-| LHCb
-| SRM endpoint entry in GOCDB
-|-
-| 120920
-| Green
-| Less Urgent
-| In Progress
-| 2016-04-19
-| 2016-05-06
-| SNO+
-| XRootD issues at RAL
 |-
 | 120810
-| Yellow
+| Amber
 | Urgent
 | In Progress
@@ Line 193: / Line 209: @@
 |-
 | 119841
-| Amber
+| Red
 | Less Urgent
 | On Hold
@@ Line 226: / Line 242: @@
 ! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment
 |-
-| 18/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
+| 25/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 95 ||
-|-
-| 19/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
-|-
-| 20/05/16 || 100 || 100 || 100 || style="background-color: lightgrey;" | 97 || 100 || 86 || 76 || Problem with test on all CEs. Looks like a CMS problem.
-|-
-| 21/05/16 || 100 || 100 || style="background-color: lightgrey;" | 28 || style="background-color: lightgrey;" | 98 || 100 || 99 || 90 || Atlas: Central monitoring problem; CMS: Single SRM test failure ("Input/output error")
-|-
-| 22/05/16 || 100 || 100 || style="background-color: lightgrey;" | 76 || 100 || 100 || 100 || 94 || Atlas: Central monitoring problem
-|-
-| 23/05/16 || 100 || 100 || 100 || style="background-color: lightgrey;" | 98 || 100 || 100 || 100 || Single SRM Test failure: User timeout.
-|-
-| 24/05/16 || 100 || 100 || 100 || 100 || style="background-color: lightgrey;" | 96 || 98 || 100 || Single SRM Test failure: (No such file or directory)
-|-
-| 25/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 |-
 | 26/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
@@ Line 249: / Line 250: @@
 | 28/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 |-
-| 29/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
+| 29/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || N/A ||
 |-
 | 30/05/16 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||

Difference between revisions of "Tier1 Operations Report 2016-06-01"

Latest revision as of 07:42, 7 June 2016

RAL Tier1 Operations Report for 1st June 2016

Navigation menu

Personal tools

Namespaces

Variants

Views

Actions

Search

Main GridPP website

Navigation

Tools