RAL Tier1 Operations Report for 22nd October 2014
Review of Issues during the week 15th to 22nd October 2014.
|
- There have been problems with three out of the four the ARC CEs since the end of last week. The process of tidying up and reporting completed jobs is taking a long time. ARC-CE01 has been declared down and is being drained as part of the investigations.
Resolved Disk Server Issues
|
- GDSS648 (LHCbUser - D1T0) failed on Monday afternoon (13th Oct). It was returned to service on 16th October after its network interface was replaced. It was put in readonly mode as a precaution and this morning (22nd) was reverted to full operation.
- GDSS720 (AtlasDataDisk - D1T0) has been completely drained pending more invasive investigations of the fault seen.
Current operational status and issues
|
- The problems reported last week with the Atlas Frontier systems was caused by the performance of the disk array that is being used temporarily by the Cronos back end database. An alternative, faster, disk array is being tested.
Ongoing Disk Server Issues
|
- GDSS763 (AtlasDataDisk - D1T0) is in production It failed on Friday (17th) and was returned to production on Monday (20th) in readonly mode. It is being drained ahead of further investigations.
Notable Changes made this last week.
|
- ARC CEs have been upgraded to version 4.2.0.
- Yesterday (14th Oct) the production FTS3 service was upgraded to version 3.2.28.
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
arc-ce01.gridpp.rl.ac.uk
|
UNSCHEDULED
|
OUTAGE
|
20/10/2014 13:45
|
23/10/2014 17:00
|
3 days, 3 hours and 15 minutes
|
Investigating problems with the ARC CE.
|
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.
Listing by category:
- Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC). (Underway).
- A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Update Castor headnodes to SL6.
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 15th and 22nd October 2014.
|
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
arc-ce01.gridpp.rl.ac.uk
|
UNSCHEDULED
|
OUTAGE
|
20/10/2014 13:45
|
23/10/2014 17:00
|
3 days, 3 hours and 15 minutes
|
Investigating problems with the ARC CE.
|
lcgwms06.gridpp.rl.ac.uk,
|
UNSCHEDULED
|
OUTAGE
|
15/10/2014 12:00
|
15/10/2014 16:37
|
4 hours and 37 minutes
|
Problems following EMI update
|
Open GGUS Tickets (Snapshot during morning of meeting)
|
GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
109399
|
Green
|
Less Urgent
|
In Progress
|
2014-10-17
|
2014-10-21
|
|
[Rod Dashboard] Issues detected at RAL-LCG2 (CE problems)
|
109360
|
Green
|
Less Urgent
|
Waiting for Reply
|
2014-10-20
|
2014-10-21
|
SNO+
|
Nagios tests failing at RAL
|
109329
|
Green
|
Less Urgent
|
In Progress
|
2014-10-14
|
2014-10-14
|
CMS
|
access to lcgvo04.gridpp.rl.ac.uk
|
109276
|
Green
|
Less Urgent
|
In Progress
|
2014-10-11
|
2014-10-13
|
CMS
|
Submissions to RAL FTS3 REST interface are failing for some users
|
109267
|
Green
|
Urgent
|
Waiting for Reply
|
2014-10-10
|
2014-10-16
|
CMS
|
possible trouble accessing pileup dataset
|
108944
|
Green
|
Urgent
|
In Progress
|
2014-10-01
|
2014-10-17
|
CMS
|
AAA access test failing at T1_UK_RAL
|
107935
|
Red
|
Less Urgent
|
On Hold
|
2014-08-27
|
2014-10-15
|
Atlas
|
BDII vs SRM inconsistent storage capacity numbers
|
107880
|
Red
|
Less Urgent
|
In Progress
|
2014-08-26
|
2014-10-15
|
SNO+
|
srmcp failure
|
106324
|
Red
|
Urgent
|
On Hold
|
2014-06-18
|
2014-10-13
|
CMS
|
pilots losing network connections at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
15/10/14 |
100 |
100 |
66.2 |
68.7 |
100 |
100 |
100 |
Problem on ARC-CEs (started previous day). Plus, for Atlas, single SRM Put test failure "HANDLING TIMEOUT"
|
16/10/14 |
100 |
100 |
91.6 |
92.0 |
100 |
100 |
100 |
CERN power cut (plus one Atlas SRM Put failure: HANDLING TIMEOUT)
|
17/10/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
18/10/14 |
100 |
100 |
99.2 |
100 |
100 |
100 |
100 |
Single SRM Put Test failure: User timeout.
|
19/10/14 |
100 |
100 |
100 |
100 |
100 |
0 |
100 |
|
20/10/14 |
100 |
100 |
99.0 |
100 |
100 |
100 |
100 |
Single SRM Put Test failure: could not open connection to srm-atlas.gridpp.rl.ac.uk
|
21/10/14 |
100 |
100 |
100 |
91.6 |
100 |
100 |
99 |
Problems with the ARC CEs.
|