RAL Tier1 Operations Report for 10th September 2014
| Review of Issues during the week 3rd to 10th September 2014.
|
- On Monday (8th Sep) there was a few minute break in network access to the Tier1 at around 5pm. Investigation showed that the primary router of our failover pair that connects the Tier1 network to the site rebooted at around 4pm. This cased two short (seconds) breaks in connectivity as the traffic failed over to the backup router then failed back. The cause of the problem an hour later, with a break in connnectivity for a few minutes, is not yet understood.
- A high rate of Atlas file access failures into/from Castor was seen during the day yesterday (9th Sep). A number of measures were taken and the problem stopped - although the underlying cause is not yet understood.
- Checks on the disk servers to be removed from the GenTape cache show a number (~300) of problematic files. These are likely to be the results of partly failed transfers into Castor in the past. These are being checked and will be followed up with the appropriate VOs.
| Resolved Disk Server Issues
|
- GDSS659 (AtlasDataDisk - D1T0) was reported last week as having problems. This server has been completedly drained, the disk controller card firmware updated and it is undergoing re-acceptance testing ahead of its re-use.
- GDSS651 (LHCbDst - D1T0) failed on Sunday morning (7th). Following investigations it was put back into service yesterday (9th). One file was reported lost from the server.
| Current operational status and issues
|
| Ongoing Disk Server Issues
|
| Notable Changes made this last week.
|
- Seven additional disk servers have been added to the disk cache for GenTape. These have 10Gbit network interfaces which enables us to withdraw some older servers with only 1 Gbit interfaces.
- CMS are now writing to the newer T10KD tapes and migration of CMS data from 'B' to 'D' tapes is underway.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgfts.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
02/09/2014 11:00
|
02/10/2014 11:00
|
30 days,
|
Service being decommissioned.
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday 30th September.
Listing by category:
- Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC).
- A new database (Oracle RAC) is being set-up that will host the Atlas3D database and be updated from CERN via a new method Oracle GoldenGate).
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Update Castor headnodes to SL6.
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
| Entries in GOC DB starting between the 3rd and 10th September 2014.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgfts.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
02/09/2014 11:00
|
02/10/2014 11:00
|
30 days,
|
Service being decommissioned.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 107935
|
Green
|
Less Urgent
|
In Progress
|
2014-08-27
|
2014-09-02
|
Atlas
|
BDII vs SRM inconsistent storage capacity numbers
|
| 107880
|
Yellow
|
Less Urgent
|
In Progress
|
2014-08-26
|
2014-09-02
|
SNO+
|
srmcp failure
|
| 106324
|
Red
|
Urgent
|
On Hold
|
2014-06-18
|
2014-08-14
|
CMS
|
pilots losing network connections at T1_UK_RAL
|
| 105405
|
Red
|
Urgent
|
On Hold
|
2014-05-14
|
2014-07-29
|
|
Please check your Vidyo router firewall configuration
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 03/09/14 |
100 |
100 |
100 |
100 |
100 |
100 |
97 |
|
| 04/09/14 |
100 |
100 |
100 |
100 |
100 |
98 |
97 |
|
| 05/09/14 |
100 |
100 |
97.0 |
100 |
100 |
100 |
99 |
Several SRM test failures.
|
| 06/09/14 |
100 |
100 |
100 |
100 |
100 |
100 |
n/a |
|
| 07/09/14 |
100 |
100 |
100 |
100 |
100 |
100 |
n/a |
|
| 08/09/14 |
100 |
100 |
100 |
100 |
100 |
95 |
88 |
|
| 09/09/14 |
100 |
100 |
94.5 |
100 |
100 |
95 |
98 |
A number of SRM test failures (PUT test failure with "Invalid argument") and Atlas Castor performace was poor at the time.
|
