Difference between revisions of "Tier1 Operations Report 2014-06-25"

From GridPP Wiki
Jump to: navigation, search
()
m ()
 
(21 intermediate revisions by 2 users not shown)
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 18th to 25th June 2014.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 18th to 25th June 2014.
 
|}
 
|}
* On Friday (13th Jun) there was a problem with one of the FTS2 front-end systems - fixed by appropriate daemon restart.
+
* On Tuesday morning (24th June) there was a crash of the database system for the Atlas SRM which failed over to another node (and was subsequently put back on the correct RAC node).
* There was a problem with cream-ce01 which started failing tests on Friday  (13th Jun). It was put into a downtime and drained out over the weekend after which its database was reset. It was returned to service on Monday (16th Jun).
+
* There were problems with the special xroot configuration for Alice following since the Castor 2.1.14 update on Tuesday (24th). These were resolved this morning (25th).
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 32: Line 32:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
* The CMS Castor stager update to version 2.1.14-13 took place yesterday (Tuesday) as planned. There were some difficulties caused by the disk server re-balancer initially after the upgrade. However, these were understood and the service was restored within the announced outage. Nevertheless there have been some ongoing problems and the service was placed in a 'warning' state in the GOC DB for the rest of the day and overnight. The problems are being followed up with the developers at CERN.
+
* There were problems with xroot following the CMS Castor stager update to version 2.1.14-13 last Tuesday (17th June). The current CMS workload exhausted the number of available xroot slots on some disk servers and then failed over to use the re-director (proxy) to source the files elsewhere. During the week the xroot settings were tuned. Significant improvement were made and the CMS Castor instance is now working OK but being closely monitored.
 +
* There have been a handful of Atlas SRM test failures which are being investigated.
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 54: Line 55:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
* Yesterday (17th June) the CMS Castor stager was updated to version 2.1.14-13.
+
* Monday (23rd June) cream-ce02 was upgraded to use EMI upgrade 14
* On Monday (16th June) Removed LHCb access from cream-ce02
+
* Tuesday (24th June) GEN Castor instance upgraded to version 2.1.14-13.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 67: Line 68:
 
|}
 
|}
 
<!-- ******* Declared in the GOC DB ******* ----->
 
<!-- ******* Declared in the GOC DB ******* ----->
* None
+
{| border=1 align=center
 +
|- bgcolor="#7c8aaf"
 +
! Service
 +
! Scheduled?
 +
! Outage/At Risk
 +
! Start
 +
! End
 +
! Duration
 +
! Reason
 +
|-
 +
| srm-lhcb.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 26/06/2014 10:00
 +
| 26/06/2014 16:00
 +
| 6 hours
 +
| LHCb Castor instance down for Castor 2.1.14 Stager Update
 +
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 81: Line 99:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
* Dates for the Castor 2.1.14 stager upgrades: GEN - Tue 24th June; LHCb - Thu 26th June; Atlas - Tues 1st July.
+
* Dates for the Castor 2.1.14 stager upgrades: LHCb - Thu 26th June; Atlas - Tues 1st July.
 
* We are starting to plan the termination of the FTS2 service now that almost all use is on FTS3.
 
* We are starting to plan the termination of the FTS2 service now that almost all use is on FTS3.
 
'''Listing by category:'''
 
'''Listing by category:'''
Line 116: Line 134:
 
! Reason
 
! Reason
 
|-
 
|-
|srm-cms.gridpp.rl.ac.uk,  
+
|srm-alice.gridpp.rl.ac.uk,  
 
| UNSCHEDULED
 
| UNSCHEDULED
 
| WARNING
 
| WARNING
| 17/06/2014 16:30
+
| 24/06/2014 17:00
| 18/06/2014 12:00
+
| 25/06/2014 12:00
| 19 hours and 30 minutes
+
| 19 hours
| Investigating some problems following the Castor 2.1.14 update of the CMS stager.
+
|Alice-disk xroot availability at risk following Castor upgrade.
 
|-
 
|-
|srm-cms.gridpp.rl.ac.uk,  
+
| Castor GEN instance: srm-alice, srm-biomed, srm-dteam, srm-hone, srm-ilc, srm-mice, srm-minos, srm-na62, srm-snoplus, srm-superb, srm-t2k.
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
| 17/06/2014 10:00
+
| 24/06/2014 09:30
| 17/06/2014 16:30
+
| 24/06/2014 17:00
| 6 hours and 30 minutes
+
| 7 hours and 30 minutes
| CMS Castor instance down for Castor 2.1.14 Stager Update.  
+
| Castor GEN instance down for Castor 2.1.14 Stager Update.
 
|-
 
|-
| cream-ce01.gridpp.rl.ac.uk, cream-ce01.gridpp.rl.ac.uk,  
+
| cream-ce02.gridpp.rl.ac.uk,  
| UNSCHEDULED
+
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
| 13/06/2014 16:30
+
| 21/06/2014 11:00
| 16/06/2014 13:11
+
| 24/06/2014 12:00
| 2 days, 20 hours and 41 minutes
+
| 3 days, 1 hour
| Problems on this CE being investigated. (All other CEs OK).
+
| EMI-3 update 14
 +
|-
 +
| srm-cms.gridpp.rl.ac.uk,
 +
| UNSCHEDULED
 +
| WARNING
 +
| 17/06/2014 16:30
 +
| 18/06/2014 12:00
 +
| 19 hours and 30 minutes
 +
| Investigating some problems following the Castor 2.1.14 update of the CMS stager.
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 155: Line 181:
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
|-
 
|-
| 106280
+
| 106481
 +
| Green
 +
| Urgent
 +
| On Hold
 +
| 2014-06-25
 +
| 2014-06-25
 +
|
 +
| [Rod Dashboard] Issues detected at RAL-LCG2
 +
|-
 +
| 106480
 
| Green
 
| Green
 
| Less Urgent
 
| Less Urgent
 
| In Progress
 
| In Progress
| 2014-06-17
+
| 2014-06-25
| 2014-06-18
+
| 2014-06-25
 +
| dteam
 +
| Publishing meaningful Castor version
 +
 
 +
|-
 +
| 106472
 +
| Green
 +
| Less Urgent
 +
| In Progress
 +
| 2014-06-25
 +
| 2014-06-25
 
| Atlas
 
| Atlas
| Lost file in RAL-LCG2
+
| Missing ATLAS file at RAL's tape instance
 +
|-
 +
| 106324
 +
| Green
 +
| Urgent
 +
| In Progress
 +
| 2014-06-18
 +
| 2014-06-23
 +
| CMS
 +
| pilots losing network connections at T1_UK_RAL
 
|-
 
|-
 
| 105571
 
| 105571
| Amber
+
| Red
 
| Less Urgent
 
| Less Urgent
 
| In Progress
 
| In Progress
Line 181: Line 235:
 
|  
 
|  
 
| please check your Vidyo router firewall configuration
 
| please check your Vidyo router firewall configuration
|-
 
| 105100
 
| Red
 
| Less Urgent
 
| In Progress
 
| 2014-05-02
 
| 2014-05-30
 
| CMS
 
| T1_UK_RAL Consistency Check (May14)
 
 
|-
 
|-
 
| 98249
 
| 98249
Line 228: Line 273:
 
| 22/06/14 || 100 || 100 || 100 || 100 || 100 || 99 || 100 ||
 
| 22/06/14 || 100 || 100 || 100 || 100 || 100 || 99 || 100 ||
 
|-
 
|-
| 23/06/14 || 100 || 100 || style="background-color: lightgrey;" | 96.7 || 100 || 100 || 100 || 99 || Couple of SRM Get test failures (SRM_FILE_BUSY)
+
| 23/06/14 || 100 || 100 ||style="background-color: lightgrey;" |96.6 || 100 || 100 || 100 || 99 || Atlas error was: 1 File was NOT copied from SRM
 
|-
 
|-
| 24/06/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
+
| 24/06/14 || 100 || style="background-color: lightgrey;" | 66 || style="background-color: lightgrey;" | 95.5 || style="background-color: lightgrey;" | 95.9 || 100 || 98 || 96 || Alice had ssues with Castor upgrades. Atlas had file not copied errors and CMS had an error stating  zero number of replicas
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->

Latest revision as of 16:20, 25 June 2014

RAL Tier1 Operations Report for 25th June 2014

Review of Issues during the week 18th to 25th June 2014.
  • On Tuesday morning (24th June) there was a crash of the database system for the Atlas SRM which failed over to another node (and was subsequently put back on the correct RAC node).
  • There were problems with the special xroot configuration for Alice following since the Castor 2.1.14 update on Tuesday (24th). These were resolved this morning (25th).
Resolved Disk Server Issues
  • None
Current operational status and issues
  • There were problems with xroot following the CMS Castor stager update to version 2.1.14-13 last Tuesday (17th June). The current CMS workload exhausted the number of available xroot slots on some disk servers and then failed over to use the re-director (proxy) to source the files elsewhere. During the week the xroot settings were tuned. Significant improvement were made and the CMS Castor instance is now working OK but being closely monitored.
  • There have been a handful of Atlas SRM test failures which are being investigated.
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • Monday (23rd June) cream-ce02 was upgraded to use EMI upgrade 14
  • Tuesday (24th June) GEN Castor instance upgraded to version 2.1.14-13.
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
srm-lhcb.gridpp.rl.ac.uk, SCHEDULED OUTAGE 26/06/2014 10:00 26/06/2014 16:00 6 hours LHCb Castor instance down for Castor 2.1.14 Stager Update
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Dates for the Castor 2.1.14 stager upgrades: LHCb - Thu 26th June; Atlas - Tues 1st July.
  • We are starting to plan the termination of the FTS2 service now that almost all use is on FTS3.

Listing by category:

  • Databases:
    • Switch LFC/FTS/3D to new Database Infrastructure.
  • Castor:
    • The Castor 2.1.14 upgrade is underway.
    • The CIP is compatible with Castor version 2.1.14. There is an issue reported by LHCb to be investigated.
  • Networking:
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
  • Fabric
    • We are phasing out the use of the software server used by the small VOs.
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
    • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.
Entries in GOC DB starting between the 18th and 25th June 2014.
Service Scheduled? Outage/At Risk Start End Duration Reason
srm-alice.gridpp.rl.ac.uk, UNSCHEDULED WARNING 24/06/2014 17:00 25/06/2014 12:00 19 hours Alice-disk xroot availability at risk following Castor upgrade.
Castor GEN instance: srm-alice, srm-biomed, srm-dteam, srm-hone, srm-ilc, srm-mice, srm-minos, srm-na62, srm-snoplus, srm-superb, srm-t2k. SCHEDULED OUTAGE 24/06/2014 09:30 24/06/2014 17:00 7 hours and 30 minutes Castor GEN instance down for Castor 2.1.14 Stager Update.
cream-ce02.gridpp.rl.ac.uk, SCHEDULED OUTAGE 21/06/2014 11:00 24/06/2014 12:00 3 days, 1 hour EMI-3 update 14
srm-cms.gridpp.rl.ac.uk, UNSCHEDULED WARNING 17/06/2014 16:30 18/06/2014 12:00 19 hours and 30 minutes Investigating some problems following the Castor 2.1.14 update of the CMS stager.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
106481 Green Urgent On Hold 2014-06-25 2014-06-25 [Rod Dashboard] Issues detected at RAL-LCG2
106480 Green Less Urgent In Progress 2014-06-25 2014-06-25 dteam Publishing meaningful Castor version
106472 Green Less Urgent In Progress 2014-06-25 2014-06-25 Atlas Missing ATLAS file at RAL's tape instance
106324 Green Urgent In Progress 2014-06-18 2014-06-23 CMS pilots losing network connections at T1_UK_RAL
105571 Red Less Urgent In Progress 2014-05-21 2014-06-02 LHCb BDII and SRM publish inconsistent storage capacity numbers
105405 Red Urgent In Progress 2014-05-14 2014-06-10 please check your Vidyo router firewall configuration
98249 Red Urgent In Progress 2013-10-21 2014-06-17 SNO+ please configure cvmfs stratum-0 for SNO+ at RAL T1
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
18/06/14 100 100 100 99.2 100 100 90 Some problems with CMS Castor following the stager upgrade led to single SRM test failure.
19/06/14 100 100 100 100 100 100 91
20/06/14 100 100 100 100 100 99 96
21/06/14 100 100 100 100 100 100 96
22/06/14 100 100 100 100 100 99 100
23/06/14 100 100 96.6 100 100 100 99 Atlas error was: 1 File was NOT copied from SRM
24/06/14 100 66 95.5 95.9 100 98 96 Alice had ssues with Castor upgrades. Atlas had file not copied errors and CMS had an error stating zero number of replicas