Difference between revisions of "Tier1 Operations Report 2015-08-05"
From GridPP Wiki
(→) |
(→) |
||
Line 9: | Line 9: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 29th July and 5th August 2015. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 29th July and 5th August 2015. | ||
|} | |} | ||
− | + | * There have been further problems with network connectivity caused by a failure of our secondary (and currently only) router. There was one as reported last week on Tuesday (28th July). At the time of last week's meeting another was ongoing. This outage lasted around five hours (???). There were problems both in getting the router back up and then in the propagation of routing information via the RIP protocol. There was then a third outage - lasting around an hour on Sunday morning (2nd August). There was a long planned scheduled intervention on 4th August when an engineer from the router vendor was present. As a result of this intervention and associated discussions we believe these problems are largely understood and we are again running with a resilient pair of network routers. | |
− | * There have been | + | |
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
<!-- *********************************************************** -----> | <!-- *********************************************************** -----> |
Revision as of 11:22, 5 August 2015
RAL Tier1 Operations Report for 5th August 2015
Review of Issues during the week 29th July and 5th August 2015. |
- There have been further problems with network connectivity caused by a failure of our secondary (and currently only) router. There was one as reported last week on Tuesday (28th July). At the time of last week's meeting another was ongoing. This outage lasted around five hours (???). There were problems both in getting the router back up and then in the propagation of routing information via the RIP protocol. There was then a third outage - lasting around an hour on Sunday morning (2nd August). There was a long planned scheduled intervention on 4th August when an engineer from the router vendor was present. As a result of this intervention and associated discussions we believe these problems are largely understood and we are again running with a resilient pair of network routers.
Resolved Disk Server Issues |
- None.
Current operational status and issues |
- We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
- The post mortem review of the network incident on the 8th April is being finalised.
- The intermittent, low-level, load-related packet loss over the OPN to CERN is still being tracked.
- There are some on-going issues for CMS. These are a problem with the Xroot (AAA) redirection accessing Castor; Slow file open times using Xroot; and poor batch job efficiencies.
Ongoing Disk Server Issues |
- None
Notable Changes made since the last meeting. |
- Three disk servers have been deployed into each of CMSDisk and LHCbDst service classes. A further disk server has been added to AtlasDataDisk to replace one that had been drained and withdrawn following a failure.
- The test of the updated worker node configuration (with grid middleware delivered via CVMFS) continues on a one whole batch of Worker Nodes.
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
Whole site | UNSCHEDULED | WARNING | 04/08/2015 15:00 | 05/08/2015 15:00 | 24 hours | Warning on Site following investigations of problem with network router. |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
- Vendor intervention on Tier1 Router scheduled for Tuesday 4th August.
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases: Castor, Atlas Frontier (LFC done)
- Castor:
- Update SRMs to new version (includes updating to SL6).
- Update the Oracle databses behind Castor to version 11.2.0.4. Will require some downtimes (See above)
- Update disk servers to SL6.
- Update to Castor version 2.1.15.
- Networking:
- Resolve problems with primary Tier1 Router. Need to schedule a roughly half-day outage for the vendor to carry out investigations.
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
- Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
- Make routing changes to allow the removal of the UKLight Router.
- Cabling/switch changes to the network in the UPS room to improve resilience.
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report. |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
Whole site | UNSCHEDULED | WARNING | 04/08/2015 15:00 | 05/08/2015 15:00 | 24 hours | Warning on Site following investigations of problem with network router. |
Whole site | SCHEDULED | OUTAGE | 04/08/2015 08:30 | 04/08/2015 15:00 | 6 hours and 30 minutes | Site Outage during investigation of problem with network router. |
Whole site | UNSCHEDULED | OUTAGE | 02/08/2015 10:00 | 02/08/2015 11:00 | 1 hour | Site was inaccessible due to a problem with the network router |
Whole site | UNSCHEDULED | WARNING | 29/07/2015 16:30 | 30/07/2015 16:30 | 24 hours | Problems on the network have been fixed. Putting whole site AT-RISK for next 24 hours. |
Whole site | UNSCHEDULED | OUTAGE | 29/07/2015 13:10 | 29/07/2015 16:51 | 3 hours and 41 minutes | Outage while we work on network problems |
Whole site | UNSCHEDULED | WARNING | 29/07/2015 11:10 | 29/07/2015 13:10 | 2 hours | Outage while we investigate network problems |
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
115336 | Green | Less Urgent | In Progress | 2015-29 | 2015-29 | OPS | [Rod Dashboard] Issue detected : hr.srce.MyProxy-Store-/ops/Role=lcgadmin@myproxy.gridpp.rl.ac.uk |
115290 | Green | Less Urgent | In Progress | 2015-07-28 | 2015-07-28 | FTS3@RAL: missing proper host names in subjectAltName of FTS agent nodes | |
114786 | Green | Less Urgent | On Hold | 2015-07-02 | 2015-07-07 | OPS | [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability |
113836 | Amber | Less Urgent | In Progress | 2015-05-20 | 2015-06-24 | GLUE 1 vs GLUE 2 mismatch in published queues | |
108944 | Red | Less Urgent | In Progress | 2014-10-01 | 2015-07-17 | CMS | AAA access test failing at T1_UK_RAL |
Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | CMS HC | Comment |
---|---|---|---|---|---|---|---|---|
29/07/15 | 77.9 | 78.0 | 78.0 | 79.0 | 73.0 | 97 | 100 | Problem with Tier1 network router. |
30/07/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
31/07/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
01/08/15 | 97.4 | 100 | 100 | 100 | 100 | 100 | 99 | SRM tests failed on Castor GEN instance. |
02/07/15 | 97.3 | 100 | 95.0 | 95.0 | 97.0 | 67 | 96 | Problem with Tier1 network router. |
03/08/15 | 100 | 100 | 100 | 94.0 | 96.0 | 93 | 97 | CMS: ARC CE test failures; LHCb: SRM test failure. |
04/07/15 | 72.9 | 73.0 | 73.0 | 73.0 | 73.0 | 100 | 56 | Planned intervention on the Tier1 Network Routers. |