Difference between revisions of "Tier1 Operations Report 2014-10-29"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
=RAL Tier1 Operations Report for 29th October 2014 (DRAFT)=
+
=RAL Tier1 Operations Report for 29th October 2014=
 
__NOTOC__
 
__NOTOC__
 
====== ======
 
====== ======
Line 10: Line 10:
 
|}
 
|}
 
* A problem reported last week, with a user filling up the spool areas on the ARC CEs has been fixed. The user was contacted (and responded) and in addition an automated process that cleans up the large log files put in place.
 
* A problem reported last week, with a user filling up the spool areas on the ARC CEs has been fixed. The user was contacted (and responded) and in addition an automated process that cleans up the large log files put in place.
 +
* There have been problems with three out of the four the ARC CEs since the end of last week. The process of tidying up and reporting completed jobs took a long time. ARC-CE01 has been declared down and was drained and re-installed last Friday (24th Oct). ARC-CE02 and ARC-CE03 were then drained over the weekend and re-installed on Monday (27th Oct).
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 20: Line 21:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
−
* GDSS673 (lhcbRawRdst - D0T1) failed around 7am on Sunday morning (26th Oct). A faulty disk drive was found. The server was checked out and firmware updated. It was returned to service during thw working day yesterday (Tuesday 28th).
+
* GDSS673 (lhcbRawRdst - D0T1) failed around 7am on Sunday morning (26th Oct). A faulty disk drive was found. The server was checked out and firmware updated. It was returned to service during the working day yesterday (Tuesday 28th). One file was declared lost from this machine.  
−
on Monday afternoon (13th Oct). It was returned to service on 16th October after its network interface was replaced. It was initially in readonly mode as a precaution. This morning (22nd) it was reverted to full (read & write) operation.
+
* As reported last week: GDSS763 (AtlasDataDisk - D1T0) had failed for the third time on around a month. This system has been completely drained and is undergoing further investigations.
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 32: Line 33:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
* There have been problems with three out of the four the ARC CEs since the end of last week. The process of tidying up and reporting completed jobs is taking a long time. ARC-CE01 has been declared down and is being drained as part of the investigations.
+
* None
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 54: Line 55:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
−
* The production FTS3 service was upgraded to version 3.2.29-1 on Monday (20th Oct).
+
* Oracle patching is ongoing with the pluto standby database patched on Monday (27th). The pluto primary is being patched today (29th).  
−
* Yesterday, Tueday 21st, an Oracle update was applied to the OGMA (Atlas 3D) database system. This morning it was switched to update the Atlas 3D information from CERN using Oracle GoldenGate, rather than Oracle Streams. This update was done during a planned Atlas test. However, it transpired that another site was had load issues with their Frontier/database system and the updated OGMA/Frontier system was returned to service very quickly after completing the change. (This alleviates a problem whereby the replacement database system for OGMA ("Cronos") was not performing well enough owing to limitations of the disk array being used temporarily within that system.)  
+
* Increased the number of jobs Alice is allowed to run to 3500. (23rd October)
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 66: Line 67:
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
|}
 
|}
−
{| border=1 align=center
+
None
−
|- bgcolor="#7c8aaf"
+
−
! Service
+
−
! Scheduled?
+
−
! Outage/At Risk
+
−
! Start
+
−
! End
+
−
! Duration
+
−
! Reason
+
−
|-
+
−
| arc-ce01.gridpp.rl.ac.uk
+
−
| UNSCHEDULED
+
−
| OUTAGE
+
−
| 20/10/2014 13:45
+
−
| 23/10/2014 17:00
+
−
| 3 days, 3 hours and 15 minutes
+
−
| Investigating problems with the ARC CE.
+
−
|}
+
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 136: Line 120:
 
! Reason
 
! Reason
 
|-
 
|-
−
| arc-ce01.gridpp.rl.ac.uk
+
|arc-ce02.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk,
 +
| UNSCHEDULED
 +
| OUTAGE
 +
| 24/10/2014 13:06
 +
| 28/10/2014 13:00
 +
| 4 days, 54 minutes
 +
|There are ongoing problems with arc-ce02 and arc-ce03. We are investigating
 +
|-
 +
|arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk,
 
| UNSCHEDULED
 
| UNSCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 20/10/2014 13:45
 
 
| 23/10/2014 17:00
 
| 23/10/2014 17:00
−
| 3 days, 3 hours and 15 minutes
+
| 24/10/2014 12:40
−
| Investigating problems with the ARC CE.
+
| 19 hours and 40 minutes
 +
|Ongoing problems with teh CE.
 
|-
 
|-
−
|lcgwms06.gridpp.rl.ac.uk,  
+
|arc-ce02.gridpp.rl.ac.uk, arc-ce02.gridpp.rl.ac.uk, arc-ce02.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk,
 +
| UNSCHEDULED
 +
| WARNING
 +
| 22/10/2014 13:00
 +
| 27/10/2014 15:00
 +
| 5 days, 3 hours
 +
|We are investigating a problem with these CEs.
 +
|-
 +
|arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk,  
 
| UNSCHEDULED
 
| UNSCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 15/10/2014 12:00
+
| 20/10/2014 13:45
−
| 15/10/2014 16:37
+
| 23/10/2014 17:00
−
| 4 hours and 37 minutes
+
| 3 days, 3 hours and 15 minutes
−
| Problems following EMI update
+
|Investigating problems with the ARC CE.
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 167: Line 167:
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
|-
 
|-
−
| 109399
+
| 109713
 
| Green
 
| Green
 
| Less Urgent
 
| Less Urgent
 
| In Progress
 
| In Progress
−
| 2014-10-17
+
| 2014-10-29
−
| 2014-10-21
+
| 2014-10-29
 
|  
 
|  
−
| [Rod Dashboard] Issues detected at RAL-LCG2 (CE problems)
+
| Add the icecube repository
 
|-
 
|-
−
| 109360
+
| 109712
 
| Green
 
| Green
−
| Less Urgent
+
| Urgent
−
| Waiting for Reply
+
| In Progress
−
| 2014-10-20
+
| 2014-10-29
−
| 2014-10-21
+
| 2014-10-29
−
| SNO+
+
| CMS
−
| Nagios tests failing at RAL
+
| Glexec exited with status 203; ...
 
|-
 
|-
−
| 109329
+
| 109708
 
| Green
 
| Green
 
| Less Urgent
 
| Less Urgent
 
| In Progress
 
| In Progress
−
| 2014-10-14
+
| 2014-10-29
−
| 2014-10-14
+
| 2014-10-29
−
| CMS
+
| Ops
−
| access to lcgvo04.gridpp.rl.ac.uk
+
| [Rod Dashboard] Issues detected at RAL-LCG2
 +
|-
 +
| 109608
 +
| Green
 +
| Urgent
 +
| Waitig for Reply
 +
| 2014-10-24
 +
| 2014-10-28
 +
| t2k
 +
| LFC denies existence of new user
 
|-
 
|-
 
| 109276
 
| 109276
Line 199: Line 208:
 
| In Progress
 
| In Progress
 
| 2014-10-11
 
| 2014-10-11
−
| 2014-10-13
+
| 2014-10-28
 
| CMS
 
| CMS
 
| Submissions to RAL FTS3 REST interface are failing for some users
 
| Submissions to RAL FTS3 REST interface are failing for some users
−
|-
 
−
| 109267
 
−
| Green
 
−
| Urgent
 
−
| Waiting for Reply
 
−
| 2014-10-10
 
−
| 2014-10-16
 
−
| CMS
 
−
| possible trouble accessing pileup dataset
 
 
|-
 
|-
 
| 108944
 
| 108944
Line 217: Line 217:
 
| In Progress
 
| In Progress
 
| 2014-10-01
 
| 2014-10-01
−
| 2014-10-17
+
| 2014-10-27
 
| CMS
 
| CMS
 
| AAA access test failing at T1_UK_RAL
 
| AAA access test failing at T1_UK_RAL
Line 235: Line 235:
 
| In Progress
 
| In Progress
 
| 2014-08-26
 
| 2014-08-26
−
| 2014-10-15
+
| 2014-10-29
 
| SNO+
 
| SNO+
 
| srmcp failure
 
| srmcp failure
Line 278: Line 278:
 
| 27/10/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||  
 
| 27/10/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||  
 
|-
 
|-
−
| 28/10/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||  
+
| 28/10/14 || 100 || 100 || style="background-color: lightgrey;" | 98.4 || 100 || 100 || 100 || 100 || Some file not copied errors on SRM.
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->

Latest revision as of 13:13, 29 October 2014

RAL Tier1 Operations Report for 29th October 2014

Review of Issues during the week 22nd to 29th October 2014.
  • A problem reported last week, with a user filling up the spool areas on the ARC CEs has been fixed. The user was contacted (and responded) and in addition an automated process that cleans up the large log files put in place.
  • There have been problems with three out of the four the ARC CEs since the end of last week. The process of tidying up and reporting completed jobs took a long time. ARC-CE01 has been declared down and was drained and re-installed last Friday (24th Oct). ARC-CE02 and ARC-CE03 were then drained over the weekend and re-installed on Monday (27th Oct).
Resolved Disk Server Issues
  • GDSS673 (lhcbRawRdst - D0T1) failed around 7am on Sunday morning (26th Oct). A faulty disk drive was found. The server was checked out and firmware updated. It was returned to service during the working day yesterday (Tuesday 28th). One file was declared lost from this machine.
  • As reported last week: GDSS763 (AtlasDataDisk - D1T0) had failed for the third time on around a month. This system has been completely drained and is undergoing further investigations.
Current operational status and issues
  • None
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • Oracle patching is ongoing with the pluto standby database patched on Monday (27th). The pluto primary is being patched today (29th).
  • Increased the number of jobs Alice is allowed to run to 3500. (23rd October)
Declared in the GOC DB

None

Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • The Oracle regular "PSU" patches will be applied to the Pluto Castor standby database system on Monday (27th Oct) and to the Pluto production database on Wednesday (29th Oct). This is expected to be a transparent intervention. The 'Pluto' database hosts the Castor Nameserver as well as the CMS and LHCb stager databases.
  • The rollout of the RIP protocol to the Tier1 routers still has to be completed.
  • First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.

Listing by category:

  • Databases:
    • Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC). (Underway).
    • A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
    • Switch LFC/3D to new Database Infrastructure.
  • Castor:
    • Update Castor headnodes to SL6.
    • Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
  • Networking:
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
    • Enable the RIP protocol for updating routing tables on the Tier1 routers.
  • Fabric
    • Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
    • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 22nd and 29th October 2014.
Service Scheduled? Outage/At Risk Start End Duration Reason
arc-ce02.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk, UNSCHEDULED OUTAGE 24/10/2014 13:06 28/10/2014 13:00 4 days, 54 minutes There are ongoing problems with arc-ce02 and arc-ce03. We are investigating
arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk, UNSCHEDULED OUTAGE 23/10/2014 17:00 24/10/2014 12:40 19 hours and 40 minutes Ongoing problems with teh CE.
arc-ce02.gridpp.rl.ac.uk, arc-ce02.gridpp.rl.ac.uk, arc-ce02.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk, arc-ce03.gridpp.rl.ac.uk, UNSCHEDULED WARNING 22/10/2014 13:00 27/10/2014 15:00 5 days, 3 hours We are investigating a problem with these CEs.
arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk, arc-ce01.gridpp.rl.ac.uk, UNSCHEDULED OUTAGE 20/10/2014 13:45 23/10/2014 17:00 3 days, 3 hours and 15 minutes Investigating problems with the ARC CE.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
109713 Green Less Urgent In Progress 2014-10-29 2014-10-29 Add the icecube repository
109712 Green Urgent In Progress 2014-10-29 2014-10-29 CMS Glexec exited with status 203; ...
109708 Green Less Urgent In Progress 2014-10-29 2014-10-29 Ops [Rod Dashboard] Issues detected at RAL-LCG2
109608 Green Urgent Waitig for Reply 2014-10-24 2014-10-28 t2k LFC denies existence of new user
109276 Green Less Urgent In Progress 2014-10-11 2014-10-28 CMS Submissions to RAL FTS3 REST interface are failing for some users
108944 Green Urgent In Progress 2014-10-01 2014-10-27 CMS AAA access test failing at T1_UK_RAL
107935 Red Less Urgent On Hold 2014-08-27 2014-10-15 Atlas BDII vs SRM inconsistent storage capacity numbers
107880 Red Less Urgent In Progress 2014-08-26 2014-10-29 SNO+ srmcp failure
106324 Red Urgent On Hold 2014-06-18 2014-10-13 CMS pilots losing network connections at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
22/10/14 100 100 100 98.9 100 99 100 Problems with the ARC CEs.
23/10/14 100 100 98.3 72.6 88.0 99 99 Some SRM test failures at around 08:00 - 09:30 local time. Compunded by some (ARC) CE test failures.
24/10/14 100 100 98.3 99.3 100 100 100 Atlas: Pair of SRM test failures (User timeout): CMS: Batch (ARC CE issues)
25/10/14 100 100 100 98.6 100 100 98 Batch (ARC CE issues)
26/10/14 100 100 100 99.4 99.1 100 n/a CMS: Batch (ARC CE issues); LHCb: SRM test failure (No such file or directory).
27/10/14 100 100 100 100 100 100 100
28/10/14 100 100 98.4 100 100 100 100 Some file not copied errors on SRM.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2014-10-29&oldid=6507"