RAL Tier1 Operations Report for 18th March 2015
Review of Issues during the week 11th to 18th March 2015.
|
- The replacemnt of the the primary Tier1 router did not fix the problem whereby the router fails after a few minutes in service.
Resolved Disk Server Issues
|
- GDSS757 (CMSDisk - D1T0) has failed multiple times this year. It has now been drained and the vendor contacted for further diagnostics. The server did fail one more time during the drain.
Current operational status and issues
|
- We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
Ongoing Disk Server Issues
|
Notable Changes made this last week.
|
- Five additional disk servers have been deployed into Castor storage for ALICE_Disk. This roughly doubles the disk space available to ALICE, bringing it up to around 390TB.
- On the CEs the pper memory limit was reduced from 5x requested memory to 3x.
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
All Castor (all SRMs)
|
SCHEDULED
|
OUTAGE
|
08/04/2015 10:00
|
08/04/2015 14:00
|
4 hours
|
Upgrade of Castor storage to version 2.1.14-15
|
Whole site
|
SCHEDULED
|
WARNING
|
19/03/2015 07:45
|
19/03/2015 08:30
|
45 minutes
|
Warning during tests of Tier1s network router. Possibility of two short (few minute) breaks in connectivity to the RAL Tier1.
|
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
- Investigate problems on the primary Tier1 router. Further tests scheduled for tomorrow morning, 19th March.
- Update Castor to 2.1-14-15 (Proposed date - Wednesday 8th April).
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4
- Castor:
- Update Castor to 2.1.14-15 (Proposed date - Wednesday 8th April).
- Update SRMs to new version (includes updating to SL6).
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Networking:
- Resolve problems with primary Tier1 Router
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
- Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
- Make routing changes to allow the removal of the UKLight Router.
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report.
|
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
Whole site
|
SCHEDULED
|
WARNING
|
12/03/2015 07:45
|
12/03/2015 08:30
|
45 minutes
|
Warning following installation of a replacement network router.
|
Whole site
|
SCHEDULED
|
WARNING
|
11/03/2015 10:30
|
11/03/2015 11:00
|
30 minutes
|
Warning on site access during firmware updates on pair of network switches.
|
Open GGUS Tickets (Snapshot during morning of meeting)
|
GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
112166
|
Green
|
Urgent
|
In Progress
|
2015-03-06
|
2015-03-11
|
Atlas
|
RAL-LCG2 DATATAPE: destination transfer errors like "This SURL does not exist in the original request"
|
111699
|
Green
|
Less Urgent
|
In Progress
|
2015-02-10
|
2015-02-27
|
Atlas
|
gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
|
109694
|
Red
|
Urgent
|
In Progress
|
2014-11-03
|
2015-03-10
|
SNO+
|
gfal-copy failing for files at RAL
|
108944
|
Red
|
Urgent
|
Waiting Reply
|
2014-10-01
|
2015-03-03
|
CMS
|
AAA access test failing at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
11/03/15 |
100 |
100 |
100 |
100 |
100 |
98 |
100 |
|
12/03/15 |
100 |
100 |
100 |
95 |
100 |
100 |
100 |
Problem with CE tests - possibly triggered by network intervention.
|
13/03/15 |
100 |
100 |
96 |
96 |
96 |
100 |
100 |
Remote monioring problem.
|
14/03/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
15/03/15 |
100 |
100 |
100 |
100 |
100 |
100 |
99 |
|
16/03/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
17/03/15 |
100 |
100 |
100 |
100 |
100 |
100 |
99 |
|