Difference between revisions of "Tier1 Operations Report 2014-05-14"

From GridPP Wiki
Jump to: navigation, search
(Created page with "=RAL Tier1 Operations Report for 14th May 2014= __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Start Review...")
 
(→‎)
 
(12 intermediate revisions by one user not shown)
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 7th to 14th May 2014.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 7th to 14th May 2014.
 
|}
 
|}
−
* Problems with "CMSDisk" in Castor reported last week have been resolved. CMS deleted files freeing up space and three more disk servers were added to the service class. (Although one of the later has subsequently failed and is currently out of service).
+
* There are some ongoing grumbly problems with the WMSs. The developers have been contacted.
−
* There was a problem on Thurdsay with the batch farm caused by a particular (biomed) user running very large jobs. This led to problems for other VOs. The jobs were killed and the user banned on the CEs. This problem recurred over the weekend as the original banning was not placed in Quattor. The user was more again banned over the weekend. A ticket about this from LHCb was raised to an 'alarm' status. This was responded to but the alert didn't page out correctly.
+
* A problem has been uncovered while investigating a missing Atlas file. Some half dozen files were lost during the draining of a disk server at the end of February. We are working to check if this is the only such instance of this and (with the developers) to understand the problem. Investigations so far suggest this is not a widespread problem. Draining of disk servers has been stopped while this investigation takes place.  
−
* There was a problem between 13:40 and 13:55 yesterday (6th May) when Argus stopped mapping DNs to users and job submissioned failed.
+
* Around ten days ago a GGUS ticket was escalated to an 'alarm' ticket by LHCb but the Tier1 pager failed to call out. This has been traced to a a problem with the e-mail from GGUS. The GGUS team were contacted. They have fixed the problem and we have tested that the alarm tickets now work again.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 22: Line 22:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
−
* GDSS758 (CMSDisk - D1T0) failed on Friday morning (2nd May). This was a a new server thta had only gone into production the previous day. Following initial investigations the server was returned to service later that day. Since then the server has been drained and is undergoing further tests.
+
* None.
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 33: Line 33:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
* We have now had repeated instances where the OPN link has not cleanly failed over to the backup link during problems with the primary.
+
* Nothing to report.
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 55: Line 55:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week.
 
|}
 
|}
−
* Deployed gdss756 - gdss758 to cmsDisk on 1st May.
+
* Testing CVMFS Client version 2.1.19 ongoing. This is now rolled out to one batch of worker nodes. So far so good.
−
* Testing CVMFS Client version 2.1.19 ongoing.
+
* pheno & enmr.eu added to list of authorized VOs on ARC CEs
 +
* FTS3 updated to version 3.2.22 this morning.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 81: Line 82:
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 13/05/2014 08:00
+
| 20/05/2014 08:00
−
| 13/05/2014 11:00
+
| 20/05/2014 11:00
 
| 3 hours
 
| 3 hours
−
| Outage of tape system for update of library controller.
+
| Outage of tape system for update of tape library controller. (Postponed from 13th May).
 
|-
 
|-
−
| All Castor (SRM) endpoints
+
| All SRMs
 
| SCHEDULED
 
| SCHEDULED
 
| WARNING
 
| WARNING
−
| 13/05/2014 08:00
+
| 20/05/2014 08:00
−
| 13/05/2014 11:00
+
| 20/05/2014 11:00
 
| 3 hours
 
| 3 hours
−
| Outage of tape system for update of library controller.  
+
| Outage of tape system for update of tape library controller. (Postponed from 13th May).
 +
|-
 +
| lcgvo08.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 15/05/2014 15:00
 +
| 15/05/2014 16:00
 +
| 1 hour
 +
| Downtime for system maintenance
 +
|-
 +
| lcgvo07.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 15/05/2014 14:00
 +
| 15/05/2014 15:00
 +
| 1 hour
 +
| Downtime for system maintenance
 +
|-
 +
| lcglb02.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 15/05/2014 13:00
 +
| 15/05/2014 14:00
 +
| 1 hour
 +
| Downtime for system maintenance
 +
|-
 +
| lcglb01.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 15/05/2014 10:45
 +
| 15/05/2014 11:45
 +
| 1 hour
 +
| Downtime for system maintenance
 +
|-
 +
| lcglb04.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 15/05/2014 09:30
 +
| 15/05/2014 10:30
 +
| 1 hour
 +
| Downtime for system maintenance
 
|-
 
|-
 
| lcgui02.gridpp.rl.ac.uk,  
 
| lcgui02.gridpp.rl.ac.uk,  
Line 115: Line 156:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
−
* Provisional dates for the Catsor 2.1.14 upgrade: Nameserver: Wed 28th May; Stagers: CMS: Tuesday 10th June; LHCb: Thursday 12th June; GEN: Tuesday 17th June; Atlas: Thursday 19th June.
+
* Provisional dates for the Castor 2.1.14 upgrade delayed to: Nameserver: Tuesday 10th June; Stagers to follow.
−
* FTS3 update planned for Wednesday 14th May (''Corrected from 14th June'').
+
* We are starting to plan the termination of the FTS2 service now that almost all use is on FTS3.
 
'''Listing by category:'''
 
'''Listing by category:'''
 
* Databases:
 
* Databases:
Line 124: Line 165:
 
* Networking:
 
* Networking:
 
** Move switches connecting recent disk servers batches ('11, '12) onto the Tier1 mesh network.
 
** Move switches connecting recent disk servers batches ('11, '12) onto the Tier1 mesh network.
−
** Update core Tier1 network and change connection to site and OPN including:
+
** Make routing changes to allow the removal of the UKLight Router.
−
*** Make routing changes to allow the removal of the UKLight Router.
+
 
* Fabric
 
* Fabric
 
** We are phasing out the use of the software server used by the small VOs.
 
** We are phasing out the use of the software server used by the small VOs.
Line 149: Line 189:
 
! Duration
 
! Duration
 
! Reason
 
! Reason
 +
|-
 +
| lcgwms06.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 14/05/2014 14:00
 +
| 14/05/2014 16:00
 +
| 2 hours
 +
| Downtime for system maintenance
 +
|-
 +
| lcgwms05.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 14/05/2014 11:30
 +
| 14/05/2014 13:30
 +
| 2 hours
 +
| Downtime for system maintenance
 +
|-
 +
| lcgfts3.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 14/05/2014 10:00
 +
| 14/05/2014 12:00
 +
| 2 hours
 +
| FTS3 service at RAL unavailable for update to version 3.2.22
 +
|-
 +
| lcgwms04.gridpp.rl.ac.uk,
 +
| SCHEDULED
 +
| OUTAGE
 +
| 14/05/2014 09:30
 +
| 14/05/2014 10:46
 +
| 1 hour and 16 minutes
 +
| Downtime for system maintenance
 
|-
 
|-
 
| lcgui02.gridpp.rl.ac.uk,  
 
| lcgui02.gridpp.rl.ac.uk,  
Line 157: Line 229:
 
| 28 days, 23 hours
 
| 28 days, 23 hours
 
| Service being decommissioned.
 
| Service being decommissioned.
−
|-
 
−
| Whole site
 
−
| SCHEDULED
 
−
| WARNING
 
−
| 30/04/2014 10:00
 
−
| 30/04/2014 12:00
 
−
| 2 hours
 
−
| RAL Tier1 site in warning state due to UPS/generator test.
 
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 180: Line 244:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 +
|-
 +
| 105374
 +
| Green
 +
| Less Urgent
 +
| In Progress
 +
| 2014-05-13
 +
| 2014-05-14
 +
| CMS
 +
| RAL: migration to tape
 +
|-
 +
| 105308
 +
| Green
 +
| Less Urgent
 +
| In Progress
 +
| 2014-05-11
 +
| 2014-05-13
 +
| Atlas
 +
| Jobs at RAL-LCG2_MCORE are failing with "Failed to open shared memory object: Permission denied"
 
|-
 
|-
 
| 105161
 
| 105161
Line 186: Line 268:
 
| In Progress
 
| In Progress
 
| 2014-05-05
 
| 2014-05-05
−
| 2014-05-06
+
| 2014-05-13
 
| H1
 
| H1
 
| hone jobs submitted into CREAM queues through lcgwms05.gridpp.rl.ac.uk & lcgwms06.gridpp.rl.ac.uk WMSs are are Ready status long time (more as 5 hours)
 
| hone jobs submitted into CREAM queues through lcgwms05.gridpp.rl.ac.uk & lcgwms06.gridpp.rl.ac.uk WMSs are are Ready status long time (more as 5 hours)
Line 193: Line 275:
 
| Green
 
| Green
 
| Urgent
 
| Urgent
−
| In Progress
+
| On Hold
 
| 2014-05-02
 
| 2014-05-02
−
| 2014-05-06
+
| 2014-05-12
 
| CMS
 
| CMS
 
| T1_UK_RAL Consistency Check (May14)
 
| T1_UK_RAL Consistency Check (May14)
−
|-
 
−
| 103197
 
−
| Red
 
−
| Less Urgent
 
−
| Waiting Reply
 
−
| 2014-04-09
 
−
| 2014-04-09
 
−
|
 
−
| RAL myproxy server and GridPP wiki
 
−
|-
 
−
| 101968
 
−
| Red
 
−
| Less Urgent
 
−
| In Progress
 
−
| 2014-03-11
 
−
| 2014-04-01
 
−
| Atlas
 
−
| RAL-LCG2_SCRATCHDISK: One dataset to delete is causing 1379 deletion errors
 
 
|-
 
|-
 
| 98249
 
| 98249
 
| Red
 
| Red
 
| Urgent
 
| Urgent
−
| In Progress
+
| Waiting Reply
 
| 2013-10-21
 
| 2013-10-21
−
| 2014-04-23
+
| 2014-05-12
 
| SNO+
 
| SNO+
 
| please configure cvmfs stratum-0 for SNO+ at RAL T1
 
| please configure cvmfs stratum-0 for SNO+ at RAL T1

Latest revision as of 10:46, 14 May 2014

RAL Tier1 Operations Report for 14th May 2014

Review of Issues during the week 7th to 14th May 2014.
  • There are some ongoing grumbly problems with the WMSs. The developers have been contacted.
  • A problem has been uncovered while investigating a missing Atlas file. Some half dozen files were lost during the draining of a disk server at the end of February. We are working to check if this is the only such instance of this and (with the developers) to understand the problem. Investigations so far suggest this is not a widespread problem. Draining of disk servers has been stopped while this investigation takes place.
  • Around ten days ago a GGUS ticket was escalated to an 'alarm' ticket by LHCb but the Tier1 pager failed to call out. This has been traced to a a problem with the e-mail from GGUS. The GGUS team were contacted. They have fixed the problem and we have tested that the alarm tickets now work again.
Resolved Disk Server Issues
  • None.
Current operational status and issues
  • Nothing to report.
Ongoing Disk Server Issues
  • None
Notable Changes made this last week.
  • Testing CVMFS Client version 2.1.19 ongoing. This is now rolled out to one batch of worker nodes. So far so good.
  • pheno & enmr.eu added to list of authorized VOs on ARC CEs
  • FTS3 updated to version 3.2.22 this morning.
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
srm-lhcb-tape.gridpp.rl.ac.uk, SCHEDULED OUTAGE 20/05/2014 08:00 20/05/2014 11:00 3 hours Outage of tape system for update of tape library controller. (Postponed from 13th May).
All SRMs SCHEDULED WARNING 20/05/2014 08:00 20/05/2014 11:00 3 hours Outage of tape system for update of tape library controller. (Postponed from 13th May).
lcgvo08.gridpp.rl.ac.uk, SCHEDULED OUTAGE 15/05/2014 15:00 15/05/2014 16:00 1 hour Downtime for system maintenance
lcgvo07.gridpp.rl.ac.uk, SCHEDULED OUTAGE 15/05/2014 14:00 15/05/2014 15:00 1 hour Downtime for system maintenance
lcglb02.gridpp.rl.ac.uk, SCHEDULED OUTAGE 15/05/2014 13:00 15/05/2014 14:00 1 hour Downtime for system maintenance
lcglb01.gridpp.rl.ac.uk, SCHEDULED OUTAGE 15/05/2014 10:45 15/05/2014 11:45 1 hour Downtime for system maintenance
lcglb04.gridpp.rl.ac.uk, SCHEDULED OUTAGE 15/05/2014 09:30 15/05/2014 10:30 1 hour Downtime for system maintenance
lcgui02.gridpp.rl.ac.uk, SCHEDULED OUTAGE 30/04/2014 14:00 29/05/2014 13:00 28 days, 23 hours Service being decommissioned.
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Provisional dates for the Castor 2.1.14 upgrade delayed to: Nameserver: Tuesday 10th June; Stagers to follow.
  • We are starting to plan the termination of the FTS2 service now that almost all use is on FTS3.

Listing by category:

  • Databases:
    • Switch LFC/FTS/3D to new Database Infrastructure.
  • Castor:
    • Castor 2.1.14 testing was largely complete, although a new minor version (2.1.14-12) will be released soon.
  • Networking:
    • Move switches connecting recent disk servers batches ('11, '12) onto the Tier1 mesh network.
    • Make routing changes to allow the removal of the UKLight Router.
  • Fabric
    • We are phasing out the use of the software server used by the small VOs.
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
    • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.
Entries in GOC DB starting between the 7th and 14th May 2014.
Service Scheduled? Outage/At Risk Start End Duration Reason
lcgwms06.gridpp.rl.ac.uk, SCHEDULED OUTAGE 14/05/2014 14:00 14/05/2014 16:00 2 hours Downtime for system maintenance
lcgwms05.gridpp.rl.ac.uk, SCHEDULED OUTAGE 14/05/2014 11:30 14/05/2014 13:30 2 hours Downtime for system maintenance
lcgfts3.gridpp.rl.ac.uk, SCHEDULED OUTAGE 14/05/2014 10:00 14/05/2014 12:00 2 hours FTS3 service at RAL unavailable for update to version 3.2.22
lcgwms04.gridpp.rl.ac.uk, SCHEDULED OUTAGE 14/05/2014 09:30 14/05/2014 10:46 1 hour and 16 minutes Downtime for system maintenance
lcgui02.gridpp.rl.ac.uk, SCHEDULED OUTAGE 30/04/2014 14:00 29/05/2014 13:00 28 days, 23 hours Service being decommissioned.
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
105374 Green Less Urgent In Progress 2014-05-13 2014-05-14 CMS RAL: migration to tape
105308 Green Less Urgent In Progress 2014-05-11 2014-05-13 Atlas Jobs at RAL-LCG2_MCORE are failing with "Failed to open shared memory object: Permission denied"
105161 Green Less Urgent In Progress 2014-05-05 2014-05-13 H1 hone jobs submitted into CREAM queues through lcgwms05.gridpp.rl.ac.uk & lcgwms06.gridpp.rl.ac.uk WMSs are are Ready status long time (more as 5 hours)
105100 Green Urgent On Hold 2014-05-02 2014-05-12 CMS T1_UK_RAL Consistency Check (May14)
98249 Red Urgent Waiting Reply 2013-10-21 2014-05-12 SNO+ please configure cvmfs stratum-0 for SNO+ at RAL T1
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
07/05/14 100 100 100 100 100 100 100
08/05/14 100 100 100 100 100 94 n/a
09/05/14 100 100 100 100 100 100 n/a
10/05/14 100 100 100 100 100 100 n/a
11/05/14 100 100 100 100 100 100 100
12/05/14 100 100 100 100 100 100 100
13/05/14 100 100 100 100 100 100 99
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2014-05-14&oldid=4377"