RAL Tier1 Operations Report for 16th September 2015
| Review of Issues during the fortnight 2nd to 16th September 2015.
|
- Following the upgrade of Castor disk servers in the disk-only service classes on Wednesday and Thursday last week it was found that some of the servers were missing correct routing information. This affected 15 servers out of just over 100 updated. The problem was traced to the ordering of the network interfaces on seen by the OS on the affected machines (which were spread across the different batches of server types) and fixed on Friday lunchtime.
| Resolved Disk Server Issues
|
| Current operational status and issues
|
- The intermittent, low-level, load-related packet loss over the OPN to CERN is still being tracked. An attempted fix (replacing the cables between the Tier1 core network and the UKLight router) had no effect.
- There are some on-going issues for CMS. There is a problem with the Xroot (AAA) redirection accessing Castor - although this has been much better understood in the last week. The poor batch job efficiencies have improved since a change to the Linux I/O scheduler on CMS disk servers a couple of weeks ago. File open times using Xroot remain slow.
| Ongoing Disk Server Issues
|
| Notable Changes made since the last meeting.
|
- The first step of the update of the Castor Oracle databases to version 11.2.0.4 was successfully carried out yesterday (15th September).
- The production FTS service was upgraded to version 3.3.1 this morning.
- Some detailed internal re-connection in the lower levels of the Tier1 network ongoing with the aim of removing the old 'core' switch. One of these required the re-connection of systems in the "UPS" room which was done during a site 'warning' on the morning of Thursday 3rd Sep.
- The roll-out of the new worker node configuration continues.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| All Castor (all SRMs)
|
SCHEDULED
|
OUTAGE
|
13/10/2015 08:00
|
13/10/2015 16:00
|
8 hours
|
Outage of All Castor instances during upgrade of Oracle back end database.
|
| All Castor (all SRMs)
|
SCHEDULED
|
WARNING
|
08/10/2015 08:30
|
08/10/2015 20:30
|
12 hours
|
Warning (At Risk) on All Castor instances during upgrade of back end Oracle database.
|
| All Castor (all SRMs)
|
SCHEDULED
|
OUTAGE
|
06/10/2015 08:00
|
06/10/2015 20:30
|
12 hours and 30 minutes
|
Outage of All Castor instances during upgrade of Oracle back end database.
|
| Whole Site.
|
SCHEDULED
|
WARNING
|
30/09/2015 07:00
|
30/09/2015 09:00
|
2 hours
|
At Risk during upgrade of network connection from the Tier1 into RAL site core network.
|
| Castor Atlas and GEN instances.
|
SCHEDULED
|
WARNING
|
22/09/2015 08:30
|
22/09/2015 20:30
|
12 hours
|
Warning (At Risk) on Atlas and GEN Castor instances during upgrade of back end Oracle database.
|
| lcgfts3.gridpp.rl.ac.uk,
|
SCHEDULED
|
WARNING
|
16/09/2015 11:00
|
16/09/2015 12:00
|
1 hour
|
Upgrade of FTS server to version 3.3.1.
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- Upgrade of remaining Castor disk servers (those in tape-backed service classes) to SL6. This will be transparent to users.
- Upgrade of the Oracle databases behind Castor to version 11.2.0.4. This is a multi-step intervention. DOngoing (further dates declared in GOC DB)
- Some detailed internal network re-configurations to be tackled now that the routers are stable. Notably:
- Change the way the UKLIGHT router connects into the Tier1 network.
- The changes to increase the bandwidth of the link from the Tier1 into the RAL internal site network will take place on the morning of 30th September.
- Extending the rollout of the new worker node configuration.
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases: Castor, Atlas Frontier (LFC done)
- Castor:
- Update SRMs to new version (includes updating to SL6).
- Update the Oracle databases behind Castor to version 11.2.0.4. Will require some downtimes (See above)
- Update disk servers to SL6 (ongoing)
- Update to Castor version 2.1.15.
- Networking:
- Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
- Make routing changes to allow the removal of the UKLight Router.
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
| Entries in GOC DB starting since the last report.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgfts3.gridpp.rl.ac.uk,
|
SCHEDULED
|
WARNING
|
16/09/2015 11:00
|
16/09/2015 12:00
|
1 hour
|
Upgrade of FTS server to version 3.3.1.
|
| Atlas and GEN Castor instances
|
UNSCHEDULED
|
WARNING
|
15/09/2015 18:15
|
16/09/2015 12:00
|
17 hours and 45 minutes
|
Warning on Atlas and GEN Castor instances following upgrade of Oracle back end database.
|
| Atlas and GEN Castor instances
|
SCHEDULED
|
OUTAGE
|
15/09/2015 08:00
|
15/09/2015 18:17
|
10 hours and 17 minutes
|
Outage of Atlas and GEN Castor instances during upgrade of Oracle back end database.
|
| Whole Site
|
SCHEDULED
|
WARNING
|
03/09/2015 08:25
|
03/09/2015 09:25
|
1 hour
|
At risk on site during brief network re-configuration. Actual change expected at 07:30 (UTC).
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 115805
|
Green
|
Less Urgent
|
In progress
|
2015-08-21
|
2015-08-21
|
SNO+
|
RAL WMS logs
|
| 115434
|
Green
|
Less Urgent
|
Waiting Reply
|
2015-08-03
|
2015-09-02
|
SNO+
|
glite-wms-job-status warning
|
| 115290
|
Green
|
Less Urgent
|
On Hold
|
2015-07-28
|
2015-07-29
|
|
FTS3@RAL: missing proper host names in subjectAltName of FTS agent nodes
|
| 108944
|
Red
|
Less Urgent
|
In Progress
|
2014-10-01
|
2015-08-26
|
CMS
|
AAA access test failing at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 02/09/15 |
100 |
100 |
98 |
100 |
100 |
96 |
98 |
Single SRM test failure.
|
| 03/09/15 |
100 |
100 |
100 |
100 |
100 |
92 |
96 |
|
| 04/09/15 |
100 |
100 |
100 |
100 |
96 |
98 |
100 |
Single SRM test failure.
|
| 05/09/15 |
100 |
100 |
98 |
100 |
100 |
96 |
100 |
Single SRM test failure.
|
| 06/09/15 |
100 |
100 |
100 |
100 |
100 |
98 |
100 |
|
| 07/09/15 |
100 |
100 |
100 |
100 |
100 |
96 |
97 |
|
| 08/09/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 09/09/15 |
100 |
100 |
100 |
100 |
100 |
98 |
100 |
|
| 10/09/15 |
100 |
100 |
98 |
100 |
100 |
96 |
100 |
Single SRM test failure.
|
| 11/09/15 |
100 |
100 |
100 |
96 |
100 |
98 |
n/a |
Single SRM test failure.
|
| 12/09/15 |
100 |
100 |
100 |
100 |
100 |
94 |
100 |
|
| 13/09/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 14/09/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 15/09/15 |
100 |
100 |
57 |
100 |
100 |
94 |
100 |
Planned Outage of Atlas & GEN Castor instances for Oracle DB upgrade.
|
