RAL Tier1 Operations Report for 25th February 2015
| Review of Issues during the week 18th to 25th February 2015.
|
- A test of the problematic Tier1 router was carried out on Thursday morning, 19th Feb.
- Yesterday (Tuesday) there was an outage of part of Castor as some racks containing disk servers (the 2011 batches) were shutdown while they were moved within the machine room to make room for new deliveries.
- There were a couple of breaks in network connectivity between 07:00 and 08:00 yesterday (Tuesday 24th Feb) while core site network switches were upgraded.
- On Wednesday (18th Feb) a single file was reportd lost to CMS. This file had been picked up by the Castor checksum checker.
| Resolved Disk Server Issues
|
- GDSS757 (CMSDisk - D1T0) failed early on Sunday morning (22nd Feb). Following checking it was returned to service the following morning.
| Current operational status and issues
|
- We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
| Ongoing Disk Server Issues
|
| Notable Changes made this last week.
|
- The 2011 disk servers were moved within the machine room to make space for new deliveries. As part of this the connections to these servers was migrated to the Tier1 mesh network.
- On Wednesday (18th Feb) the Argus policy was updated to support the MICE reco role.
- A Castor namesever box has been set-up to enable queries against Castor metadata to be made without affecting the throughput of production work.
- A system has been set-up to provide Atlas with Castor information that is not supplied by the SRM.
None
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- Investigate problems on the primary Tier1 router. Discussions with the vendor are ongoing.
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4
- Castor:
- Update SRMs to new version (includes updating to SL6).
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Update Castor to 2.1-14-latest.
- Networking:
- Resolve problems with primary Tier1 Router
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
| Entries in GOC DB starting since the last report.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| Many Castor services (srm-alice, srm-atlas, srm-biomed, srm-dteam, srm-ilc, srm-lhcb-tape, srm-lhcb, srm-mice, srm-na62, srm-pheno, srm-snoplus, srm-superb, srm-t2k)
|
SCHEDULED
|
OUTAGE
|
24/02/2015 08:30
|
24/02/2015 17:06
|
8 hours and 36 minutes
|
Outage of some parts of the Castor storage while disk servers are moved to make space for new equipment.
|
| Whole site
|
SCHEDULED
|
WARNING
|
19/02/2015 07:45
|
19/02/2015 08:30
|
45 minutes
|
Warning during test of network router. We expect a 5 minute break in connectivity to site sometime during this window.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 111856
|
Green
|
Less Urgent
|
In Progress
|
2015-02-19
|
2015-02-24
|
LHCb
|
Stalled LHCb jobs over night
|
| 111699
|
Green
|
Less Urgent
|
In Progress
|
2015-02-10
|
2015-02-18
|
Atlas
|
gLExec hammercloud jobs keep failing at RAL-LCG2 & RALPP
|
| 109694
|
Red
|
Urgent
|
Waiting Reply
|
2014-11-03
|
2015-02-24
|
SNO+
|
gfal-copy failing for files at RAL
|
| 108944
|
Red
|
Urgent
|
In Progress
|
2014-10-01
|
2015-02-24
|
CMS
|
AAA access test failing at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 18/02/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 19/02/15 |
100 |
100 |
98.0 |
100 |
100 |
98 |
97 |
Single SRM test failure
|
| 20/02/15 |
100 |
100 |
98.0 |
100 |
100 |
100 |
100 |
Single SRM test failure
|
| 21/02/15 |
100 |
100 |
100 |
100 |
100 |
100 |
99 |
|
| 22/02/15 |
100 |
100 |
98.0 |
100 |
100 |
100 |
99 |
Single SRM test failure
|
| 23/02/15 |
100 |
100 |
100 |
100 |
100 |
97 |
99 |
|
| 24/02/15 |
100 |
100 |
62.0 |
58.0 |
59.0 |
94 |
92 |
Planned move of disk server racks.
|
