Difference between revisions of "Tier1 Operations Report 2016-04-13"

From GridPP Wiki
Jump to: navigation, search
(Created page with "==RAL Tier1 Operations Report for 6th April 2016== __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Start Rev...")
 
 
(11 intermediate revisions by one user not shown)
Line 1: Line 1:
−
==RAL Tier1 Operations Report for 6th April 2016==
+
==RAL Tier1 Operations Report for 13th April 2016==
 
__NOTOC__
 
__NOTOC__
 
====== ======
 
====== ======
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 6th to 13th April 2016.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 6th to 13th April 2016.
 
|}
 
|}
−
* As reported last week there had been a network problem on the evening of Tuesday 30th March between around 18:30 and 21:00. One of the RAL site routers had a problem and the link between it and the primary of our Tier1 router pair was flapping. There should have been an automatic failover to our secondary router - but owing to a configuration error (since understood) that did not happen. Staff attended on site to force the failover which resolved the problem. Following the fix to the RAL site router we reverted to using the Primary Tier1 router this morning (6th April).
+
* None
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 20: Line 20:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
−
* GDSS635 (AtlasTape - D0T1) was taken out of production on the 26th March following a crash. There were no files awaiting migration to tape. It was returned to service on Thursday (31st March) although no specific cause for the crash had been identified.
+
* GDSS771 (LHCbDst - D1T0) crashed on Saturday (9th April). It has since crashed a number of times. It was put back in service read-only the following day and is being drained ahead of more extensive investigations.
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 33: Line 33:
 
* There is a problem seen by LHCb of a low but persistent rate of failure when copying the results of batch jobs to Castor. There is also a further problem that sometimes occurs when these (failed) writes are attempted to storage at other sites. A recent modification has improved, but not completed fixed this.
 
* There is a problem seen by LHCb of a low but persistent rate of failure when copying the results of batch jobs to Castor. There is also a further problem that sometimes occurs when these (failed) writes are attempted to storage at other sites. A recent modification has improved, but not completed fixed this.
 
* The intermittent, low-level, load-related packet loss seen over external connections is still being tracked. Likewise we have been working to understand some remaining low level of packet loss seen within a part of our Tier1 network.
 
* The intermittent, low-level, load-related packet loss seen over external connections is still being tracked. Likewise we have been working to understand some remaining low level of packet loss seen within a part of our Tier1 network.
−
* The draining of Castor disk servers is very slow. We need to drain a few old servers to provide spares - this being part of the deployment plan and server lifecycle. This does not yet impact services but is being followed up.  
+
* The draining of Castor disk servers for Atlas is very slow. We need to drain a few old servers to provide spares - this being part of the deployment plan and server lifecycle. This does not yet impact services but is being followed up.  
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 44: Line 44:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues
 
|}
 
|}
−
* GDSS635 (AtlasTape - D0T1) crashed this morning. It is currently under investigation.
+
* GDSS635 (AtlasTape - D0T1) crashed on the 6th April. It is still being worked on. There are no files awaiting migration to tape on the server so this has no impact on file availability.
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ************************************************************* ----->
 
<!-- ************************************************************* ----->
Line 55: Line 55:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
|}
 
|}
−
* A load balancer (a pair of systems running "HAProxy") has been introduced in front of the "test" FTS3 instance which is used by Atlas. At present this just handles some 20% of the requests to the service. It will gradually ramp up to handle all requests.
+
* None
−
* Eight disk servers, each with 111TBytes storage, have been  deployed to AtlasdataDisk.
+
−
* Atlas have now been swung over to writing new data to the T10KD drives.
+
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 82: Line 80:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
−
* The Castor 2.1.15 update is pending. Testing has shown a database related performance issue which is being followed up. We await successful resolution of that problem and completion of testing before scheduling. In the meantime we plan to carry out the update to the Castor SRMs.
+
* The Castor 2.1.15 update is pending. Testing has shown a database related performance issue which is being followed up. We await successful resolution of that problem and completion of testing before scheduling. Following advice from the developers we will not upgrade the SRMs before the Castor 2.1.15 upgrade.
−
* Decommissioning of "GEN Scratch" storage in Castor.
+
* Decommissioning of "GEN Scratch" storage in Castor. (Formally announced by EGI broadcast).
 
'''Listing by category:'''
 
'''Listing by category:'''
 
* Databases:
 
* Databases:
Line 90: Line 88:
 
** Update SRMs to new version (includes updating to SL6).
 
** Update SRMs to new version (includes updating to SL6).
 
** Update to Castor version 2.1.15.
 
** Update to Castor version 2.1.15.
−
** Migration of data from T10KC to T10KD tapes (affects Atlas & LHCb data).
+
** Migration of data from T10KC to T10KD tapes (Affects Atlas & LHCb data).
 
* Networking:
 
* Networking:
 
** Replace the UKLight Router. Then upgrade the 'bypass' link to the RAL border routers to 2*10Gbit.
 
** Replace the UKLight Router. Then upgrade the 'bypass' link to the RAL border routers to 2*10Gbit.
Line 96: Line 94:
 
** Firmware updates on remaining EMC disk arrays (Castor, LFC)
 
** Firmware updates on remaining EMC disk arrays (Castor, LFC)
 
* Grid Services
 
* Grid Services
−
** A Load Balancer (HAProxy) will be used in front of the FTS service.
+
** Once the use of the Load Balancer (HAProxy) has been proven for the test FTS service it will be extended to other services.
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ***************End Advanced warning for other interventions*************** ----->
 
<!-- ************************************************************************** ----->
 
<!-- ************************************************************************** ----->
Line 122: Line 120:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 +
|-
 +
| 120810
 +
| Green
 +
| Urgent
 +
| In Progress
 +
| 2016-04-13
 +
| 2016-04-13
 +
| Biomed
 +
| Decommissioning of SE srm-biomed.gridpp.rl.ac.uk - forbid write access for biomed users
 +
|-
 +
| 120797
 +
| Green
 +
| Top priority
 +
| In Progress
 +
| 2016-04-12
 +
| 2016-04-14
 +
| Atlas
 +
| Downloads failing from gdss612 in RAL-LCG2_SCRATCHDISK "The requested service is not available at the moment"
 
|-
 
|-
 
| 120624
 
| 120624
Line 131: Line 147:
 
| CMS
 
| CMS
 
| Consistency Check for T1_UK_RAL
 
| Consistency Check for T1_UK_RAL
 +
|-
 +
| 120604
 +
| Red
 +
| Urgent
 +
| Reopened
 +
| 2016-04-12
 +
| 2016-04-12
 +
| Atlas
 +
| Source file size mismatch at RAL-LCG2 SCRATCHDISK
 
|-
 
|-
 
| 120350
 
| 120350
Line 174: Line 199:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! CMS HC !! Comment
−
|-
 
−
| 30/03/16 || 100 || 100 || style="background-color: lightgrey;" | 97 || 100 || 100 || N/A || 96 || Single SRM Test failure (Unable to schedule transfer )
 
−
|-
 
−
| 31/03/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||
 
−
|-
 
−
| 01/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||
 
−
|-
 
−
| 02/04/16 || 100 || 100 || 100 || 100 || style="background-color: lightgrey;" | 96 || N/A || 100 || Single SRM Test failure on List:  [SRM_INVALID_PATH] No such file or directory
 
−
|-
 
−
| 03/04/16 || 100 || 100 || 100 || style="background-color: lightgrey;" | 70 || 100 || N/A || N/A || Problem with test submission (also affected other sites).
 
−
|-
 
−
| 04/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||
 
−
|-
 
−
| 05/04/16 || 100 || 100 || style="background-color: lightgrey;" | 97 || 100 || 100 || N/A || 100 || Single SRM Test failure on GET (User timeout)
 
 
|-
 
|-
 
| 06/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
| 06/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
|-
 
|-
−
| 07/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
+
| 07/04/16 || 100 || 100 || 100 || style="background-color: lightgrey;" | 97 || 100 || N/A || 100 || Problem with test submission. Other sites also affected.
 
|-
 
|-
 
| 08/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
| 08/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
Line 197: Line 208:
 
| 09/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
| 09/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
|-
 
|-
−
| 10/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
+
| 10/04/16 || 100 || 100 || style="background-color: lightgrey;" | 98 || style="background-color: lightgrey;" | 53 || style="background-color: lightgrey;" | 96 || N/A || 100 || Atlas & LHCb: Single SRM Test failures; CMS: Problem with test submission. Other sites also affected.
 
|-
 
|-
 
| 11/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
| 11/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
Line 203: Line 214:
 
| 12/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
| 12/04/16 || 100 || 100 || 100 || 100 || 100 || N/A || 100 ||  
 
|-
 
|-
−
 
−
 
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->

Latest revision as of 09:46, 20 April 2016

RAL Tier1 Operations Report for 13th April 2016

Review of Issues during the week 6th to 13th April 2016.
  • None
Resolved Disk Server Issues
  • GDSS771 (LHCbDst - D1T0) crashed on Saturday (9th April). It has since crashed a number of times. It was put back in service read-only the following day and is being drained ahead of more extensive investigations.
Current operational status and issues
  • There is a problem seen by LHCb of a low but persistent rate of failure when copying the results of batch jobs to Castor. There is also a further problem that sometimes occurs when these (failed) writes are attempted to storage at other sites. A recent modification has improved, but not completed fixed this.
  • The intermittent, low-level, load-related packet loss seen over external connections is still being tracked. Likewise we have been working to understand some remaining low level of packet loss seen within a part of our Tier1 network.
  • The draining of Castor disk servers for Atlas is very slow. We need to drain a few old servers to provide spares - this being part of the deployment plan and server lifecycle. This does not yet impact services but is being followed up.
Ongoing Disk Server Issues
  • GDSS635 (AtlasTape - D0T1) crashed on the 6th April. It is still being worked on. There are no files awaiting migration to tape on the server so this has no impact on file availability.
Notable Changes made since the last meeting.
  • None
Declared in the GOC DB

None

Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • The Castor 2.1.15 update is pending. Testing has shown a database related performance issue which is being followed up. We await successful resolution of that problem and completion of testing before scheduling. Following advice from the developers we will not upgrade the SRMs before the Castor 2.1.15 upgrade.
  • Decommissioning of "GEN Scratch" storage in Castor. (Formally announced by EGI broadcast).

Listing by category:

  • Databases:
    • Switch LFC/3D to new Database Infrastructure.
  • Castor:
    • Update SRMs to new version (includes updating to SL6).
    • Update to Castor version 2.1.15.
    • Migration of data from T10KC to T10KD tapes (Affects Atlas & LHCb data).
  • Networking:
    • Replace the UKLight Router. Then upgrade the 'bypass' link to the RAL border routers to 2*10Gbit.
  • Fabric
    • Firmware updates on remaining EMC disk arrays (Castor, LFC)
  • Grid Services
    • Once the use of the Load Balancer (HAProxy) has been proven for the test FTS service it will be extended to other services.
Entries in GOC DB starting since the last report.
  • None
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
120810 Green Urgent In Progress 2016-04-13 2016-04-13 Biomed Decommissioning of SE srm-biomed.gridpp.rl.ac.uk - forbid write access for biomed users
120797 Green Top priority In Progress 2016-04-12 2016-04-14 Atlas Downloads failing from gdss612 in RAL-LCG2_SCRATCHDISK "The requested service is not available at the moment"
120624 Green Urgent Waiting Reply 2016-04-05 2016-04-05 CMS Consistency Check for T1_UK_RAL
120604 Red Urgent Reopened 2016-04-12 2016-04-12 Atlas Source file size mismatch at RAL-LCG2 SCRATCHDISK
120350 Green Less Urgent In Progress 2016-03-22 2016-04-05 LSST Enable LSST at RAL
119841 Green Less Urgent In Progress 2016-03-01 2016-03-22 LHCb HTTP support for lcgcadm04.gridpp.rl.ac.uk
117683 Green Less Urgent On Hold 2015-11-18 2016-04-05 CASTOR at RAL not publishing GLUE 2
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
06/04/16 100 100 100 100 100 N/A 100
07/04/16 100 100 100 97 100 N/A 100 Problem with test submission. Other sites also affected.
08/04/16 100 100 100 100 100 N/A 100
09/04/16 100 100 100 100 100 N/A 100
10/04/16 100 100 98 53 96 N/A 100 Atlas & LHCb: Single SRM Test failures; CMS: Problem with test submission. Other sites also affected.
11/04/16 100 100 100 100 100 N/A 100
12/04/16 100 100 100 100 100 N/A 100
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2016-04-13&oldid=11246"