Difference between revisions of "Tier1 Operations Report 2014-07-02"

From GridPP Wiki
Jump to: navigation, search
(Created page with "=RAL Tier1 Operations Report for 2nd July 2014= __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Start Review...")
 
(→‎)
 
(8 intermediate revisions by one user not shown)
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 25th June to 2nd July 2014.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 25th June to 2nd July 2014.
 
|}
 
|}
−
* On Tuesday morning (24th June) there was a crash of the database system for the Atlas SRM which failed over to another node (and was subsequently put back on the correct RAC node).
+
* There were Atlas SRM test failures during last week which were traced to load on the Atlas Castor system during searches for dark data. Less intrusive ways of carrying out tasks such as this are being investigated.
−
* There were problems with the special xroot configuration for Alice following since the Castor 2.1.14 update on Tuesday (24th). These were resolved this morning (25th).
+
* There was a problem with one (of the three) Top-BDII systems which was unavailable for around 40 minutes from midday yesterday (1st July).
 +
* A UPS/Generator load test was successfully carried out this morning (Wednesday 2nd July).
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 32: Line 33:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
* There were problems with xroot following the CMS Castor stager update to version 2.1.14-13 last Tuesday (17th June). The current CMS workload exhausted the number of available xroot slots on some disk servers and then failed over to use the re-director (proxy) to source the files elsewhere. During the week the xroot settings were tuned. Significant improvement were made and the CMS Castor instance is now working OK but being closely monitored.
+
* We continue to monitor closely the performance of xroot access to CMS Castor following the upgrade on the 17th June. Performance is generally good although the current CMS workflow causes some problems with hot files.
−
* There have been a handful of Atlas SRM test failures which are being investigated.
+
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 55: Line 55:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
−
* Monday (23rd June) cream-ce02 was upgraded to use EMI upgrade 14
+
* Thursday (26th June) LHCb Castor instance upgraded to version 2.1.14-13.
−
* Tuesday (24th June) GEN Castor instance upgraded to version 2.1.14-13.
+
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 78: Line 77:
 
! Reason
 
! Reason
 
|-
 
|-
−
| srm-lhcb.gridpp.rl.ac.uk,  
+
|srm-atlas.gridpp.rl.ac.uk,  
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 26/06/2014 10:00
+
| 08/07/2014 06:00
−
| 26/06/2014 16:00
+
| 09/07/2014 12:00
−
| 6 hours
+
| 1 day, 6 hours
−
| LHCb Castor instance down for Castor 2.1.14 Stager Update
+
| Atlas Castor instance down for Castor 2.1.14 Stager Update
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 99: Line 98:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
−
* Dates for the Castor 2.1.14 stager upgrades: LHCb - Thu 26th June; Atlas - Tues 1st July.
+
* Date for the final Castor 2.1.14 stager upgrades of Atlas: Tuesday 8th July.
−
* We are starting to plan the termination of the FTS2 service now that almost all use is on FTS3.
+
* We are planning the termination of the FTS2 service (proposed for 2nd September) now that almost all use is on FTS3.
 
'''Listing by category:'''
 
'''Listing by category:'''
 
* Databases:
 
* Databases:
Line 106: Line 105:
 
* Castor:
 
* Castor:
 
** The Castor 2.1.14 upgrade is underway.  
 
** The Castor 2.1.14 upgrade is underway.  
−
** The CIP is compatible with Castor version 2.1.14. There is an issue reported by LHCb to be investigated.
 
 
* Networking:
 
* Networking:
 
** Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
 
** Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
Line 134: Line 132:
 
! Reason
 
! Reason
 
|-
 
|-
−
|srm-alice.gridpp.rl.ac.uk,
+
| Whole Site
−
| UNSCHEDULED
+
| SCHEDULED
 
| WARNING
 
| WARNING
−
| 24/06/2014 17:00
+
| 02/07/2014 10:00
−
| 25/06/2014 12:00
+
| 02/07/2014 11:00
−
| 19 hours
+
| 1 hour
−
|Alice-disk xroot availability at risk following Castor upgrade.
+
| RAL Tier1 site in warning state due to UPS/generator test.
 
|-
 
|-
−
| Castor GEN instance: srm-alice, srm-biomed, srm-dteam, srm-hone, srm-ilc, srm-mice, srm-minos, srm-na62, srm-snoplus, srm-superb, srm-t2k.  
+
| srm-lhcb.gridpp.rl.ac.uk
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 24/06/2014 09:30
+
| 26/06/2014 10:00
−
| 24/06/2014 17:00
+
| 26/06/2014 11:19
−
| 7 hours and 30 minutes
+
| 1 hour and 19 minutes
−
| Castor GEN instance down for Castor 2.1.14 Stager Update.
+
| LHCb Castor instance down for Castor 2.1.14 Stager Update
 
|-
 
|-
−
| cream-ce02.gridpp.rl.ac.uk,
+
| srm-lhcb.gridpp.rl.ac.uk
−
| SCHEDULED
+
| UNSCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 21/06/2014 11:00
+
| 26/06/2014 09:00
−
| 24/06/2014 12:00
+
| 26/06/2014 10:00
−
| 3 days, 1 hour
+
| 1 hour
−
| EMI-3 update 14
+
| Starting LHCb Castor instance Castor 2.1.14 Stager Update one hour earlier.
 
|-
 
|-
−
| srm-cms.gridpp.rl.ac.uk,
+
| srm-alice.gridpp.rl.ac.uk
 
| UNSCHEDULED
 
| UNSCHEDULED
 
| WARNING
 
| WARNING
−
| 17/06/2014 16:30
+
| 24/06/2014 17:00
−
| 18/06/2014 12:00
+
| 25/06/2014 12:00
−
| 19 hours and 30 minutes
+
| 19 hours
−
| Investigating some problems following the Castor 2.1.14 update of the CMS stager.
+
| Alice-disk xroot availability at risk following Castor upgrade.
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 180: Line 178:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
−
|-
 
−
| 106481
 
−
| Green
 
−
| Urgent
 
−
| On Hold
 
−
| 2014-06-25
 
−
| 2014-06-25
 
−
|
 
−
| [Rod Dashboard] Issues detected at RAL-LCG2
 
 
|-
 
|-
 
| 106480
 
| 106480
 
| Green
 
| Green
 
| Less Urgent
 
| Less Urgent
−
| In Progress
+
| Waiting Reply
−
| 2014-06-25
+
 
| 2014-06-25
 
| 2014-06-25
 +
| 2014-06-30
 
| dteam
 
| dteam
 
| Publishing meaningful Castor version
 
| Publishing meaningful Castor version
−
 
−
|-
 
−
| 106472
 
−
| Green
 
−
| Less Urgent
 
−
| In Progress
 
−
| 2014-06-25
 
−
| 2014-06-25
 
−
| Atlas
 
−
| Missing ATLAS file at RAL's tape instance
 
 
|-
 
|-
 
| 106324
 
| 106324
−
| Green
+
| Yellow
 
| Urgent
 
| Urgent
 
| In Progress
 
| In Progress
 
| 2014-06-18
 
| 2014-06-18
−
| 2014-06-23
+
| 2014-07-01
 
| CMS
 
| CMS
 
| pilots losing network connections at T1_UK_RAL
 
| pilots losing network connections at T1_UK_RAL
Line 223: Line 202:
 
| In Progress
 
| In Progress
 
| 2014-05-21
 
| 2014-05-21
−
| 2014-06-02
+
| 2014-06-30
 
| LHCb
 
| LHCb
 
| BDII and SRM publish inconsistent storage capacity numbers
 
| BDII and SRM publish inconsistent storage capacity numbers
Line 230: Line 209:
 
| Red
 
| Red
 
| Urgent
 
| Urgent
−
| In Progress
+
| On Hold
 
| 2014-05-14
 
| 2014-05-14
−
| 2014-06-10
+
| 2014-07-01
 
|  
 
|  
 
| please check your Vidyo router firewall configuration
 
| please check your Vidyo router firewall configuration
Line 263: Line 242:
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 25/06/14 || 100 || 100 || style="background-color: lightgrey;" |94.8 || 100 || 100 || 96 || 98 ||
+
| 25/06/14 || 100 || 100 || style="background-color: lightgrey;" |94.8 || 100 || 100 || 96 || 98 || Several SUM test failures (Invalid Argument).
 
|-
 
|-
−
| 26/06/14 || 100 || 100 || style="background-color: lightgrey;" |90.6 || style="background-color: lightgrey;" |95.8 || style="background-color: lightgrey;" |92.6 || 90 || 100 ||
+
| 26/06/14 || 100 || 100 || style="background-color: lightgrey;" |90.6 || style="background-color: lightgrey;" |95.8 || style="background-color: lightgrey;" |92.6 || 90 || 100 || LHCb Castor Stager 2.1.14 upgrade; Atlas: Several SRM test failures; CMS: Single SRM Put test failure.
 
|-
 
|-
 
| 27/06/14 || 100 || 100 || 100 || 100 || 100 || 99 || 100 ||
 
| 27/06/14 || 100 || 100 || 100 || 100 || 100 || 99 || 100 ||

Latest revision as of 10:15, 2 July 2014

RAL Tier1 Operations Report for 2nd July 2014

Review of Issues during the week 25th June to 2nd July 2014.
  • There were Atlas SRM test failures during last week which were traced to load on the Atlas Castor system during searches for dark data. Less intrusive ways of carrying out tasks such as this are being investigated.
  • There was a problem with one (of the three) Top-BDII systems which was unavailable for around 40 minutes from midday yesterday (1st July).
  • A UPS/Generator load test was successfully carried out this morning (Wednesday 2nd July).
Resolved Disk Server Issues
  • None
Current operational status and issues
  • We continue to monitor closely the performance of xroot access to CMS Castor following the upgrade on the 17th June. Performance is generally good although the current CMS workflow causes some problems with hot files.
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • Thursday (26th June) LHCb Castor instance upgraded to version 2.1.14-13.
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
srm-atlas.gridpp.rl.ac.uk, SCHEDULED OUTAGE 08/07/2014 06:00 09/07/2014 12:00 1 day, 6 hours Atlas Castor instance down for Castor 2.1.14 Stager Update
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Date for the final Castor 2.1.14 stager upgrades of Atlas: Tuesday 8th July.
  • We are planning the termination of the FTS2 service (proposed for 2nd September) now that almost all use is on FTS3.

Listing by category:

  • Databases:
    • Switch LFC/FTS/3D to new Database Infrastructure.
  • Castor:
    • The Castor 2.1.14 upgrade is underway.
  • Networking:
    • Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
  • Fabric
    • We are phasing out the use of the software server used by the small VOs.
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
    • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.
Entries in GOC DB starting between the 25th June and 2nd July 2014.
Service Scheduled? Outage/At Risk Start End Duration Reason
Whole Site SCHEDULED WARNING 02/07/2014 10:00 02/07/2014 11:00 1 hour RAL Tier1 site in warning state due to UPS/generator test.
srm-lhcb.gridpp.rl.ac.uk SCHEDULED OUTAGE 26/06/2014 10:00 26/06/2014 11:19 1 hour and 19 minutes LHCb Castor instance down for Castor 2.1.14 Stager Update
srm-lhcb.gridpp.rl.ac.uk UNSCHEDULED OUTAGE 26/06/2014 09:00 26/06/2014 10:00 1 hour Starting LHCb Castor instance Castor 2.1.14 Stager Update one hour earlier.
srm-alice.gridpp.rl.ac.uk UNSCHEDULED WARNING 24/06/2014 17:00 25/06/2014 12:00 19 hours Alice-disk xroot availability at risk following Castor upgrade.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
106480 Green Less Urgent Waiting Reply 2014-06-25 2014-06-30 dteam Publishing meaningful Castor version
106324 Yellow Urgent In Progress 2014-06-18 2014-07-01 CMS pilots losing network connections at T1_UK_RAL
105571 Red Less Urgent In Progress 2014-05-21 2014-06-30 LHCb BDII and SRM publish inconsistent storage capacity numbers
105405 Red Urgent On Hold 2014-05-14 2014-07-01 please check your Vidyo router firewall configuration
98249 Red Urgent In Progress 2013-10-21 2014-06-17 SNO+ please configure cvmfs stratum-0 for SNO+ at RAL T1
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
25/06/14 100 100 94.8 100 100 96 98 Several SUM test failures (Invalid Argument).
26/06/14 100 100 90.6 95.8 92.6 90 100 LHCb Castor Stager 2.1.14 upgrade; Atlas: Several SRM test failures; CMS: Single SRM Put test failure.
27/06/14 100 100 100 100 100 99 100
28/06/14 100 100 100 100 100 96 100
29/06/14 100 100 100 100 100 98 98
30/06/14 100 100 100 100 100 94 100
01/07/14 100 100 100 100 100 100 100
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2014-07-02&oldid=5041"