Difference between revisions of "Tier1 Operations Report 2015-09-16"
From GridPP Wiki
(Created page with "==RAL Tier1 Operations Report for 16th September 2015== __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Star...") |
(→) |
||
(10 intermediate revisions by one user not shown) | |||
Line 9: | Line 9: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the fortnight 2nd to 16th September 2015. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the fortnight 2nd to 16th September 2015. | ||
|} | |} | ||
− | * | + | * On the 9th September it was found that xrootd transfers for ALICE had not been working since the disk server SL6 upgrade on 26/27 August. This was due to an incorrect xrootd configuration. |
+ | * At the end of last week there was a problem with the Atlas Castor instance - which a few times became unresponsive for some tens of minutes. | ||
+ | * There have been some problems with the production FTS service which has been struggling under very high load. Yesterday (15th Sep) a significant portion of the Atlas traffic being handled by this server was moved to our "test" FTS instance. We are monitoring the service to check that as the backlog reduces the systems performance does recover. | ||
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
<!-- *********************************************************** -----> | <!-- *********************************************************** -----> | ||
Line 31: | Line 33: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | ||
|} | |} | ||
− | * The intermittent, low-level, load-related packet loss over | + | * The intermittent, low-level, load-related packet loss seen over external connections is still being tracked. |
− | * | + | * Over recent weeks there have been some work on the long-standing CMS issues. Batch job efficiencies now look good (e.g. taking last month's figures). A new server has been provided for the CMS Xroot (AAA) redirection to Castor but the problems remain. File open times using Xroot remain slow but this is a less significant problem. |
<!-- ***********End Current operational status and issues*********** -----> | <!-- ***********End Current operational status and issues*********** -----> | ||
<!-- *************************************************************** -----> | <!-- *************************************************************** -----> | ||
Line 54: | Line 56: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting. | ||
|} | |} | ||
− | * Castor | + | * The first step of the update of the Castor Oracle databases to version 11.2.0.4 was successfully carried out yesterday (15th September). |
− | * Some detailed internal re-connection in the lower levels of the Tier1 network | + | * The production FTS service was upgraded to version 3.3.1 this morning. |
+ | * Some detailed internal re-connection in the lower levels of the Tier1 network ongoing with the aim of removing the old 'core' switch. One of these required the re-connection of systems in the "UPS" room which was done during a site 'warning' on the morning of Thursday 3rd Sep. | ||
* The roll-out of the new worker node configuration continues. | * The roll-out of the new worker node configuration continues. | ||
<!-- *************End Notable Changes made this last week************** -----> | <!-- *************End Notable Changes made this last week************** -----> | ||
Line 77: | Line 80: | ||
! Reason | ! Reason | ||
|- | |- | ||
− | | Castor | + | | All Castor (all SRMs) |
| SCHEDULED | | SCHEDULED | ||
| OUTAGE | | OUTAGE | ||
Line 85: | Line 88: | ||
| Outage of All Castor instances during upgrade of Oracle back end database. | | Outage of All Castor instances during upgrade of Oracle back end database. | ||
|- | |- | ||
− | | All Castor | + | | All Castor (all SRMs) |
| SCHEDULED | | SCHEDULED | ||
| WARNING | | WARNING | ||
Line 93: | Line 96: | ||
| Warning (At Risk) on All Castor instances during upgrade of back end Oracle database. | | Warning (At Risk) on All Castor instances during upgrade of back end Oracle database. | ||
|- | |- | ||
− | | All Castor | + | | All Castor (all SRMs) |
| SCHEDULED | | SCHEDULED | ||
| OUTAGE | | OUTAGE | ||
Line 101: | Line 104: | ||
| Outage of All Castor instances during upgrade of Oracle back end database. | | Outage of All Castor instances during upgrade of Oracle back end database. | ||
|- | |- | ||
− | | Castor Atlas | + | | Whole Site. |
+ | | SCHEDULED | ||
+ | | WARNING | ||
+ | | 30/09/2015 07:00 | ||
+ | | 30/09/2015 09:00 | ||
+ | | 2 hours | ||
+ | | At Risk during upgrade of network connection from the Tier1 into RAL site core network. | ||
+ | |- | ||
+ | | Castor Atlas and GEN instances. | ||
| SCHEDULED | | SCHEDULED | ||
| WARNING | | WARNING | ||
Line 109: | Line 120: | ||
| Warning (At Risk) on Atlas and GEN Castor instances during upgrade of back end Oracle database. | | Warning (At Risk) on Atlas and GEN Castor instances during upgrade of back end Oracle database. | ||
|- | |- | ||
− | | | + | | lcgfts3.gridpp.rl.ac.uk, |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
| SCHEDULED | | SCHEDULED | ||
| WARNING | | WARNING | ||
− | | | + | | 16/09/2015 11:00 |
− | | | + | | 16/09/2015 12:00 |
| 1 hour | | 1 hour | ||
− | | | + | | Upgrade of FTS server to version 3.3.1. |
|} | |} | ||
<!-- **********************End GOC DB Entries************************** -----> | <!-- **********************End GOC DB Entries************************** -----> | ||
Line 139: | Line 142: | ||
<!-- ******* still to be formally scheduled and/or announced ******* -----> | <!-- ******* still to be formally scheduled and/or announced ******* -----> | ||
* Upgrade of remaining Castor disk servers (those in tape-backed service classes) to SL6. This will be transparent to users. | * Upgrade of remaining Castor disk servers (those in tape-backed service classes) to SL6. This will be transparent to users. | ||
− | * Upgrade of the Oracle databases behind Castor to version 11.2.0.4. | + | * Upgrade of the Oracle databases behind Castor to version 11.2.0.4. The first step of this multi-step upgrade has been carried out. Further dates are declared in the GOC DB. |
− | * Some detailed internal network | + | * Some detailed internal network re-configurations to be tackled now that the routers are stable. Notably: |
− | + | ||
** Change the way the UKLIGHT router connects into the Tier1 network. | ** Change the way the UKLIGHT router connects into the Tier1 network. | ||
+ | ** Changes to increase the bandwidth of the link from the Tier1 into the RAL internal site network. This will take place on the morning of 30th September. | ||
* Extending the rollout of the new worker node configuration. | * Extending the rollout of the new worker node configuration. | ||
'''Listing by category:''' | '''Listing by category:''' | ||
Line 156: | Line 159: | ||
** Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit. | ** Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit. | ||
** Make routing changes to allow the removal of the UKLight Router. | ** Make routing changes to allow the removal of the UKLight Router. | ||
− | |||
* Fabric | * Fabric | ||
** Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC) | ** Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC) | ||
Line 179: | Line 181: | ||
! Reason | ! Reason | ||
|- | |- | ||
− | | | + | |lcgfts3.gridpp.rl.ac.uk, |
+ | | SCHEDULED | ||
+ | | WARNING | ||
+ | | 16/09/2015 11:00 | ||
+ | | 16/09/2015 12:00 | ||
+ | | 1 hour | ||
+ | |Upgrade of FTS server to version 3.3.1. | ||
+ | |- | ||
+ | | Atlas and GEN Castor instances | ||
| UNSCHEDULED | | UNSCHEDULED | ||
| WARNING | | WARNING | ||
− | | | + | | 15/09/2015 18:15 |
− | | | + | | 16/09/2015 12:00 |
− | | | + | | 17 hours and 45 minutes |
− | | | + | | Warning on Atlas and GEN Castor instances following upgrade of Oracle back end database. |
|- | |- | ||
− | | | + | | Atlas and GEN Castor instances |
| SCHEDULED | | SCHEDULED | ||
− | | | + | | OUTAGE |
− | | | + | | 15/09/2015 08:00 |
− | | | + | | 15/09/2015 18:17 |
− | | | + | | 10 hours and 17 minutes |
− | | | + | | Outage of Atlas and GEN Castor instances during upgrade of Oracle back end database. |
|- | |- | ||
− | | | + | | Whole Site |
| SCHEDULED | | SCHEDULED | ||
| WARNING | | WARNING | ||
− | | | + | | 03/09/2015 08:25 |
− | | | + | | 03/09/2015 09:25 |
− | | | + | | 1 hour |
− | | | + | | At risk on site during brief network re-configuration. Actual change expected at 07:30 (UTC). |
|} | |} | ||
<!-- **********************End GOC DB Entries************************** -----> | <!-- **********************End GOC DB Entries************************** -----> | ||
Line 218: | Line 228: | ||
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject | ! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject | ||
|- | |- | ||
− | | | + | | 116180 |
| Green | | Green | ||
| Less Urgent | | Less Urgent | ||
| In progress | | In progress | ||
− | | 2015- | + | | 2015-09-14 |
− | | 2015- | + | | 2015-09-14 |
− | | | + | | DiRAC |
− | | | + | | Failing fts transmissions |
|- | |- | ||
− | | | + | | 116094 |
| Green | | Green | ||
− | | | + | | Very Urgent |
− | | | + | | In Progress |
− | | 2015- | + | | 2015-09-07 |
− | | 2015-09- | + | | 2015-09-15 |
− | | | + | | Atlas |
− | | | + | | Many thousands file transfers failed to NL, ND, IT, UK in the last 4 hours |
|- | |- | ||
| 115290 | | 115290 | ||
Line 244: | Line 254: | ||
| | | | ||
| FTS3@RAL: missing proper host names in subjectAltName of FTS agent nodes | | FTS3@RAL: missing proper host names in subjectAltName of FTS agent nodes | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
|} | |} | ||
<!-- **********************End GGUS Tickets************************** -----> | <!-- **********************End GGUS Tickets************************** -----> | ||
Line 286: | Line 287: | ||
| 09/09/15 || 100 || 100 || 100 || 100 || 100 || 98 || 100 || | | 09/09/15 || 100 || 100 || 100 || 100 || 100 || 98 || 100 || | ||
|- | |- | ||
− | | 10/09/15 || 100 || 100 || 98 || 100 || 100 || 96 || 100 || Single SRM test failure. | + | | 10/09/15 || 100 || 100 || style="background-color: lightgrey;" | 98 || 100 || 100 || 96 || 100 || Single SRM test failure. |
|- | |- | ||
| 11/09/15 || 100 || 100 || 100 || style="background-color: lightgrey;" | 96 || 100 || 98 || n/a || Single SRM test failure. | | 11/09/15 || 100 || 100 || 100 || style="background-color: lightgrey;" | 96 || 100 || 98 || n/a || Single SRM test failure. |
Latest revision as of 12:24, 16 September 2015
RAL Tier1 Operations Report for 16th September 2015
Review of Issues during the fortnight 2nd to 16th September 2015. |
- On the 9th September it was found that xrootd transfers for ALICE had not been working since the disk server SL6 upgrade on 26/27 August. This was due to an incorrect xrootd configuration.
- At the end of last week there was a problem with the Atlas Castor instance - which a few times became unresponsive for some tens of minutes.
- There have been some problems with the production FTS service which has been struggling under very high load. Yesterday (15th Sep) a significant portion of the Atlas traffic being handled by this server was moved to our "test" FTS instance. We are monitoring the service to check that as the backlog reduces the systems performance does recover.
Resolved Disk Server Issues |
- None
Current operational status and issues |
- The intermittent, low-level, load-related packet loss seen over external connections is still being tracked.
- Over recent weeks there have been some work on the long-standing CMS issues. Batch job efficiencies now look good (e.g. taking last month's figures). A new server has been provided for the CMS Xroot (AAA) redirection to Castor but the problems remain. File open times using Xroot remain slow but this is a less significant problem.
Ongoing Disk Server Issues |
- None
Notable Changes made since the last meeting. |
- The first step of the update of the Castor Oracle databases to version 11.2.0.4 was successfully carried out yesterday (15th September).
- The production FTS service was upgraded to version 3.3.1 this morning.
- Some detailed internal re-connection in the lower levels of the Tier1 network ongoing with the aim of removing the old 'core' switch. One of these required the re-connection of systems in the "UPS" room which was done during a site 'warning' on the morning of Thursday 3rd Sep.
- The roll-out of the new worker node configuration continues.
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
All Castor (all SRMs) | SCHEDULED | OUTAGE | 13/10/2015 08:00 | 13/10/2015 16:00 | 8 hours | Outage of All Castor instances during upgrade of Oracle back end database. |
All Castor (all SRMs) | SCHEDULED | WARNING | 08/10/2015 08:30 | 08/10/2015 20:30 | 12 hours | Warning (At Risk) on All Castor instances during upgrade of back end Oracle database. |
All Castor (all SRMs) | SCHEDULED | OUTAGE | 06/10/2015 08:00 | 06/10/2015 20:30 | 12 hours and 30 minutes | Outage of All Castor instances during upgrade of Oracle back end database. |
Whole Site. | SCHEDULED | WARNING | 30/09/2015 07:00 | 30/09/2015 09:00 | 2 hours | At Risk during upgrade of network connection from the Tier1 into RAL site core network. |
Castor Atlas and GEN instances. | SCHEDULED | WARNING | 22/09/2015 08:30 | 22/09/2015 20:30 | 12 hours | Warning (At Risk) on Atlas and GEN Castor instances during upgrade of back end Oracle database. |
lcgfts3.gridpp.rl.ac.uk, | SCHEDULED | WARNING | 16/09/2015 11:00 | 16/09/2015 12:00 | 1 hour | Upgrade of FTS server to version 3.3.1. |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
- Upgrade of remaining Castor disk servers (those in tape-backed service classes) to SL6. This will be transparent to users.
- Upgrade of the Oracle databases behind Castor to version 11.2.0.4. The first step of this multi-step upgrade has been carried out. Further dates are declared in the GOC DB.
- Some detailed internal network re-configurations to be tackled now that the routers are stable. Notably:
- Change the way the UKLIGHT router connects into the Tier1 network.
- Changes to increase the bandwidth of the link from the Tier1 into the RAL internal site network. This will take place on the morning of 30th September.
- Extending the rollout of the new worker node configuration.
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases: Castor, Atlas Frontier (LFC done)
- Castor:
- Update SRMs to new version (includes updating to SL6).
- Update the Oracle databases behind Castor to version 11.2.0.4. Will require some downtimes (See above)
- Update disk servers to SL6 (ongoing)
- Update to Castor version 2.1.15.
- Networking:
- Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
- Make routing changes to allow the removal of the UKLight Router.
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report. |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
lcgfts3.gridpp.rl.ac.uk, | SCHEDULED | WARNING | 16/09/2015 11:00 | 16/09/2015 12:00 | 1 hour | Upgrade of FTS server to version 3.3.1. |
Atlas and GEN Castor instances | UNSCHEDULED | WARNING | 15/09/2015 18:15 | 16/09/2015 12:00 | 17 hours and 45 minutes | Warning on Atlas and GEN Castor instances following upgrade of Oracle back end database. |
Atlas and GEN Castor instances | SCHEDULED | OUTAGE | 15/09/2015 08:00 | 15/09/2015 18:17 | 10 hours and 17 minutes | Outage of Atlas and GEN Castor instances during upgrade of Oracle back end database. |
Whole Site | SCHEDULED | WARNING | 03/09/2015 08:25 | 03/09/2015 09:25 | 1 hour | At risk on site during brief network re-configuration. Actual change expected at 07:30 (UTC). |
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
116180 | Green | Less Urgent | In progress | 2015-09-14 | 2015-09-14 | DiRAC | Failing fts transmissions |
116094 | Green | Very Urgent | In Progress | 2015-09-07 | 2015-09-15 | Atlas | Many thousands file transfers failed to NL, ND, IT, UK in the last 4 hours |
115290 | Green | Less Urgent | On Hold | 2015-07-28 | 2015-07-29 | FTS3@RAL: missing proper host names in subjectAltName of FTS agent nodes |
Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | CMS HC | Comment |
---|---|---|---|---|---|---|---|---|
02/09/15 | 100 | 100 | 98 | 100 | 100 | 96 | 98 | Single SRM test failure. |
03/09/15 | 100 | 100 | 100 | 100 | 100 | 92 | 96 | |
04/09/15 | 100 | 100 | 100 | 100 | 96 | 98 | 100 | Single SRM test failure. |
05/09/15 | 100 | 100 | 98 | 100 | 100 | 96 | 100 | Single SRM test failure. |
06/09/15 | 100 | 100 | 100 | 100 | 100 | 98 | 100 | |
07/09/15 | 100 | 100 | 100 | 100 | 100 | 96 | 97 | |
08/09/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
09/09/15 | 100 | 100 | 100 | 100 | 100 | 98 | 100 | |
10/09/15 | 100 | 100 | 98 | 100 | 100 | 96 | 100 | Single SRM test failure. |
11/09/15 | 100 | 100 | 100 | 96 | 100 | 98 | n/a | Single SRM test failure. |
12/09/15 | 100 | 100 | 100 | 100 | 100 | 94 | 100 | |
13/09/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
14/09/15 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
15/09/15 | 100 | 100 | 57 | 100 | 100 | 94 | 100 | Planned Outage of Atlas & GEN Castor instances for Oracle DB upgrade. |