RAL Tier1 Operations Report for 10th September 2014
Review of Issues during the week 3rd to 10th September 2014.
|
- A network switch failed overnight Friday-Saturday (29/30 Aug). Staff attended on site and the immediate problem was resolved. However, further problems were found with a number of VMs providing services that took some time to fix. Not all services were affected - the site (except Castor) was declared down for around 6 hours on Saturday.
Resolved Disk Server Issues
|
- GDSS659 (AtlasDataDisk - D1T0) was reported last week as having problems. This server has been completedly drained, the disk controller card firmware updated and is undrgoing re-acceptance testing ahead of its re-use.
- GDSS651 (LHCbDst - D1T0) failed on Sunday morning (7th). Following investigations it was put back into service yesterday (9th). One file was reported lost from the server.
Current operational status and issues
|
- Discrepancies were found in some of the Castor database tables and columns. The Castor team are considering options with regard to fixing these. The issue has no operational impact.
- We are still investigating xroot access to CMS Castor following the upgrade on the 17th June. The service has improved but there may still be work to be done.
Ongoing Disk Server Issues
|
Notable Changes made this last week.
|
- The FTS2 service was ended yesterday, 2nd September. The servers were shutdown.
- The Software Server that was used by the smaller VOs has been stopped.
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
lcgfts.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
02/09/2014 11:00
|
02/10/2014 11:00
|
30 days,
|
Service being decommissioned.
|
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday 23rd September.
Listing by category:
- Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, Atlas3D).
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes; migration of GEN from 'A' to 'D' tapes.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room - date to be decided.
Entries in GOC DB starting between the 3rd and 10th September 2014.
|
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
lcgfts.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
02/09/2014 11:00
|
02/10/2014 11:00
|
30 days,
|
Service being decommissioned.
|
Open GGUS Tickets (Snapshot during morning of meeting)
|
GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
107935
|
Green
|
Less Urgent
|
In Progress
|
2014-08-27
|
2014-09-02
|
Atlas
|
BDII vs SRM inconsistent storage capacity numbers
|
107880
|
Yellow
|
Less Urgent
|
In Progress
|
2014-08-26
|
2014-09-02
|
SNO+
|
srmcp failure
|
106324
|
Red
|
Urgent
|
On Hold
|
2014-06-18
|
2014-08-14
|
CMS
|
pilots losing network connections at T1_UK_RAL
|
105405
|
Red
|
Urgent
|
On Hold
|
2014-05-14
|
2014-07-29
|
|
Please check your Vidyo router firewall configuration
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
03/09/14 |
100 |
100 |
100 |
100 |
100 |
100 |
97 |
|
04/09/14 |
100 |
100 |
100 |
100 |
100 |
98 |
97 |
|
05/09/14 |
100 |
100 |
97.0 |
100 |
100 |
100 |
99 |
Several SRM test failures.
|
06/09/14 |
100 |
100 |
100 |
100 |
100 |
100 |
n/a |
|
07/09/14 |
100 |
100 |
100 |
100 |
100 |
100 |
n/a |
|
08/09/14 |
100 |
100 |
100 |
100 |
100 |
95 |
88 |
|
09/09/14 |
100 |
100 |
94.5 |
100 |
100 |
95 |
98 |
A number of SRM test failures (PUT test failure with "Invalid argument") and Atlas Castor performace was poor at the time.
|