RAL Tier1 Operations Report for 15th October 2014
Review of Issues during the week 8th to 15th October 2014.
|
- As reported last week there was a problem with one of the site BDIIs that caused a problem for SAM tests against CEs through Tuesday/Wednesday last week (20/9 - 01/10).
- On Monday (6th October) performance problems were experienced with Castor - particularly affecting Atlas. It was found that the battery for the memory cache in the disk array hosting the Castor databases had failed. The resulted in much slower performance of the disk array, which in turn impacted Castor performance. A reconfiguration was made to lighten the load on the disk array and restore performance. One of the two databases (Neptune, which hosts the Atlas and GEN SRM databases) was moved to the standby database system. This required an outage of the Castor Atlas and GEN instances which lasted around 2 hours. The standby is, under normal operation, kept up to date using Oracle 'DataGuard' which replicates the updates applied to the primary database onto the standby. At the time of writing this report we are awaiting an engineer to come and fix the battery problem.
- Two files were declared lost to ALICE, both from AliceDisk. These wre picked up by the Castor checksum checker.
Resolved Disk Server Issues
|
Current operational status and issues
|
- The problems reported last week with the Atlas Frontier systems was caused by the performance of the disk array that is being used temporarily by the Cronos back end database. An alternative, faster, disk array is being tested.
- The Castor databases are currently running in a temporary configuration while we await an engineer to come and fix the problem with the battery powering the cache on a disk array.
Ongoing Disk Server Issues
|
- GDSS648 (LHCbUser - D1T0) failed on monday afternoon (13th Oct). The problem is in the networking and investigations coare ongoing.
Notable Changes made this last week.
|
- On Monday (6th October) access to the two cream CEs (cream-ce01, cream-ce02) was modified to only accept ALICE, dteam and ops jobs plus snoplus.snolab.ca.
- Oracle patches (PSU) applied to the production Neptune database (Castor Atlas & GEN) on Thursday (2nd October).
- ALICE have made successful tests running batch work through the ARC CEs.
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- First quarter 2015: Circuit testing of the remaining (i.e. non-UPS) circuits in the machine room.
Listing by category:
- Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC). (Underway).
- A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Update Castor headnodes to SL6.
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 8th and 15th October 2014.
|
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
lcgfts3.gridpp.rl.ac.uk,
|
SCHEDULED
|
WARNING
|
14/10/2014 12:00
|
14/10/2014 13:00
|
1 hour
|
Upgrade FTS3 servers to version 3.2.28
|
Atlas and GEN Castor instances (srm-alice, srm-atlas, srm-biomed, srm-dteam, srm-hone, srm-ilc, srm-mice, srm-minos, srm-na62, srm-snoplus, srm-superb, srm-t2k
|
SCHEDULED
|
OUTAGE
|
14/10/2014 10:00
|
14/10/2014 12:00
|
2 hours
|
Outage of Castor instances ATLAS and GEN (Alice plus non-LHC VOs) during database re-organisation.
|
arc-ce02, arc-ce03, arc-ce04
|
SCHEDULED
|
WARNING
|
13/10/2014 11:00
|
13/10/2014 12:00
|
1 hour
|
Upgrade of ARC CEs to version 4.2.0 (Delayed from 9th October).
|
lcgwms05.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
12/10/2014 09:00
|
14/10/2014 17:00
|
2 days, 8 hours
|
update to EMI 3.20
|
lcgwms04.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
09/10/2014 14:00
|
10/10/2014 16:35
|
1 day, 2 hours and 35 minutes
|
draining before EMI 3 update 20 update
|
arc-ce02, arc-ce03, arc-ce04
|
SCHEDULED
|
WARNING
|
09/10/2014 10:00
|
09/10/2014 11:00
|
1 hour
|
Upgrade of ARC CEs to version 4.2.0
|
All Castor
|
UNSCHEDULED
|
WARNING
|
07/10/2014 15:00
|
08/10/2014 16:27
|
1 day, 1 hour and 27 minutes
|
Extending warning while databases running with degraded RAID battery
|
Open GGUS Tickets (Snapshot during morning of meeting)
|
GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
108944
|
Green
|
Urgent
|
In Progress
|
2014-10-01
|
2014-10-01
|
CMS
|
AAA access test failing at T1_UK_RAL
|
108845
|
Green
|
Urgent
|
In Progress
|
2014-09-27
|
2014-10-07
|
Atlas
|
RAL-LCG2: Source connection timeout plus globus_ftp_client error
|
108546
|
Amber
|
Less Urgent
|
In Progress
|
2014-09-16
|
2014-09-22
|
Atlas
|
RAL-LCG2_HIMEM_SL6: production jobs failed
|
107935
|
Red
|
Less Urgent
|
In Progress
|
2014-08-27
|
2014-10-07
|
Atlas
|
BDII vs SRM inconsistent storage capacity numbers
|
107880
|
Red
|
Less Urgent
|
Waiting Reply
|
2014-08-26
|
2014-10-08
|
SNO+
|
srmcp failure
|
106324
|
Red
|
Urgent
|
In Progress
|
2014-06-18
|
2014-09-23
|
CMS
|
pilots losing network connections at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
08/10/14 |
100 |
100 |
100 |
100 |
100 |
91 |
100 |
|
09/10/14 |
100 |
100 |
100 |
100 |
100 |
95 |
100 |
|
10/10/14 |
100 |
100 |
100 |
100 |
100 |
94 |
94 |
|
11/10/14 |
100 |
100 |
100 |
100 |
100 |
95 |
93 |
|
12/10/14 |
100 |
100 |
100 |
100 |
100 |
98 |
98 |
|
13/10/14 |
100 |
100 |
100 |
100 |
100 |
99 |
98 |
|
14/10/14 |
100 |
100 |
88.6 |
96.5 |
100 |
91 |
99 |
For Atlas: An outage of two hours during scheduled Castor DB re-configuration. Atlas & CMS: Evening problems with ARC CEs.
|