Difference between revisions of "Tier1 Operations Report 2015-07-15"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(8 intermediate revisions by one user not shown)
Line 9: Line 9:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 8th to 15th July 2015.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 8th to 15th July 2015.
 
|}
 
|}
−
* A couple of issues that have emerged as part of the testing of the new worker node configuration have been identified and solved. One of these was seen by LHCb (and reported last week), the other was seen by Atlas.
+
* On Thursday (9th July) Atlas reported a problem with a big backlog of FTS transfers from Glasgow to RAL. Investigations showed that there was some packet loss seen between Glasgow and RAL. The FTS had automatically tuned the number of concurrent file transfers down. Forcing the FTS system to use  more typical settings unblocked the system. The backlog had cleared by the following morning.
−
* A problem transferring some files from FNAL to RAL (reported last week) has been resolved by FNAL.
+
−
* The 'OPS' SUM test has been failing at RAL, as for many other sites, owing to a problem with the test itself.
+
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 57: Line 55:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
|}
 
|}
−
* A UPS/Generator load test was made successfully last Wednesday (1st July).
+
* The test of the updated worker node configuration (with grid middleware delivered via CVMFS) has been extended to a whole batch of WNs.
−
* A test of using the Castor re-balancer more aggressively was made. However, these showed some internal operational problems and it has been tuned back down again.
+
* Following completion of the roll-out of the modified Castor database procedure we have started adjusting the length of a 'wait' to study the effect on file open times. Initial change for LHCb Castor instance this morning (Wed 15th July).
−
* The change a database procedure to speeds up file open times within Castor was made to the GEN instance on Wednesday afternoon (1st July). This change has now been rolled out to all Castor instances.
+
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 80: Line 77:
 
! Reason
 
! Reason
 
|-
 
|-
−
| arc-ce05.gridpp.rl.ac.uk
+
| Whole site
 
| SCHEDULED
 
| SCHEDULED
 
| OUTAGE
 
| OUTAGE
−
| 10/06/2015 12:00
+
| 04/08/2015 08:30
−
| 08/07/2015 12:00
+
| 04/08/2015 15:00
−
| 28 days,
+
| 6 hours and 30 minutes
−
| This particular ARC CE will be decommisioned. (Four other ARC CEs remain in production).
+
|Site Outage during investigation of problem with network router.
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 101: Line 98:
 
|}
 
|}
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
−
* Vendor intervention on Tier1 Router provisionally scheduled for Tuesday 4th August.
+
* Vendor intervention on Tier1 Router scheduled for Tuesday 4th August.
 
'''Listing by category:'''
 
'''Listing by category:'''
 
* Databases:
 
* Databases:
Line 129: Line 126:
 
| style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report.
 
| style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report.
 
|}
 
|}
−
{| border=1 align=center
+
* None
−
|- bgcolor="#7c8aaf"
+
−
! Service
+
−
! Scheduled?
+
−
! Outage/At Risk
+
−
! Start
+
−
! End
+
−
! Duration
+
−
! Reason
+
−
|-
+
−
| Whole site
+
−
| SCHEDULED
+
−
| WARNING
+
−
| 01/07/2015 10:00
+
−
| 01/07/2015 11:30
+
−
| 1 hour and 30 minutes
+
−
| Warning on site during quarterly UPS/Generator load test.
+
−
|-
+
−
| arc-ce05.gridpp.rl.ac.uk
+
−
| SCHEDULED
+
−
| OUTAGE
+
−
| 10/06/2015 12:00
+
−
| 08/07/2015 12:00
+
−
| 28 days,
+
−
| This particular ARC CE will be decommisioned. (Four other ARC CEs remain in production).
+
−
|}
+
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 169: Line 141:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 +
|-
 +
| 114957
 +
| Green
 +
| Less Urgent
 +
| In Progress
 +
| 2015-07-09
 +
| 2015-07-10
 +
| Atlas
 +
| Lpad-RAL-LCG2 frontier squid in red
 
|-
 
|-
 
| 114786
 
| 114786
Line 178: Line 159:
 
| OPS
 
| OPS
 
| [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability
 
| [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability
−
|-
 
−
| 113910
 
−
| Green
 
−
| Less Urgent
 
−
| Waiting Reply
 
−
| 2015-05-26
 
−
| 2015-06-23
 
−
| SNO+
 
−
| RAL data staging
 
 
|-
 
|-
 
| 113836
 
| 113836
Line 226: Line 198:
 
| 08/07/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
| 08/07/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
|-
 
|-
−
| 09/07/15 || 100 || 100 || 100 || 90.0 || 100 || 100 || 100 || CE test failures. (CVMFS issue affecting CMS globally.)
+
| 09/07/15 || 100 || 100 || 100 || style="background-color: lightgrey;" | 90.0 || 100 || 100 || 100 || CE test failures. (CVMFS issue affecting CMS globally.)
 
|-
 
|-
 
| 10/07/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
| 10/07/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
Line 236: Line 208:
 
| 13/07/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
| 13/07/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
 
|-
 
|-
−
| 14/07/15 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
+
| 14/07/15 || 100 || 100 || 100 || style="background-color: lightgrey;" | 92.0 || style="background-color: lightgrey;" | 95.0 || 100 || 99 || CMS: Two SRM test failures; LHCb: Single SRM test failure.
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
 
<!-- *********************************************************************** ----->
 
<!-- *********************************************************************** ----->

Latest revision as of 11:39, 15 July 2015

RAL Tier1 Operations Report for 15th July 2015

Review of Issues during the week 8th to 15th July 2015.
  • On Thursday (9th July) Atlas reported a problem with a big backlog of FTS transfers from Glasgow to RAL. Investigations showed that there was some packet loss seen between Glasgow and RAL. The FTS had automatically tuned the number of concurrent file transfers down. Forcing the FTS system to use more typical settings unblocked the system. The backlog had cleared by the following morning.
Resolved Disk Server Issues
  • GDSS680 (AtlasDataDisk - D1T0) had a read-only filesystem on Friday afternoon (10th July). The filesystem was repaired and the server returned to production the following morning. A faulty disk has since been replaced.
Current operational status and issues
  • We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
  • The post mortem review of the network incident on the 8th April is being finalised.
  • The intermittent, low-level, load-related packet loss over the OPN to CERN is still being tracked.
Ongoing Disk Server Issues
  • GDSS657 (lhcbRawRdst, D0T1) failed on Saturday (11th July). Six files that had not yet been copied to tape were found to be corrupt. Remaining un-migrated files have been copied to another disk server.
Notable Changes made since the last meeting.
  • The test of the updated worker node configuration (with grid middleware delivered via CVMFS) has been extended to a whole batch of WNs.
  • Following completion of the roll-out of the modified Castor database procedure we have started adjusting the length of a 'wait' to study the effect on file open times. Initial change for LHCb Castor instance this morning (Wed 15th July).
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
Whole site SCHEDULED OUTAGE 04/08/2015 08:30 04/08/2015 15:00 6 hours and 30 minutes Site Outage during investigation of problem with network router.
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • Vendor intervention on Tier1 Router scheduled for Tuesday 4th August.

Listing by category:

  • Databases:
    • Switch LFC/3D to new Database Infrastructure.
    • Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases: Castor, Atlas Frontier (LFC done)
  • Castor:
    • Update SRMs to new version (includes updating to SL6).
    • Update the Oracle databses behind Castor to version 11.2.0.4. Will require some downtimes (See above)
    • Update disk servers to SL6.
    • Update to Castor version 2.1.15.
  • Networking:
    • Resolve problems with primary Tier1 Router. Need to schedule a roughly half-day outage for the vendor to carry out investigations.
    • Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
    • Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
    • Make routing changes to allow the removal of the UKLight Router.
    • Cabling/switch changes to the network in the UPS room to improve resilience.
  • Fabric
    • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report.
  • None
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
114957 Green Less Urgent In Progress 2015-07-09 2015-07-10 Atlas Lpad-RAL-LCG2 frontier squid in red
114786 Green Less Urgent On Hold 2015-07-02 2015-07-07 OPS [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability
113836 Yellow Less Urgent In Progress 2015-05-20 2015-06-24 GLUE 1 vs GLUE 2 mismatch in published queues
108944 Red Less Urgent In Progress 2014-10-01 2015-06-16 CMS AAA access test failing at T1_UK_RAL
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
08/07/15 100 100 100 100 100 100 100
09/07/15 100 100 100 90.0 100 100 100 CE test failures. (CVMFS issue affecting CMS globally.)
10/07/15 100 100 100 100 100 100 100
11/07/15 100 100 100 100 100 100 100
12/07/15 100 100 100 100 100 100 100
13/07/15 100 100 100 100 100 100 100
14/07/15 100 100 100 92.0 95.0 100 99 CMS: Two SRM test failures; LHCb: Single SRM test failure.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2015-07-15&oldid=9037"