RAL Tier1 Operations Report for 1st July 2015
Review of Issues during the week 24th June to 1st July 2015.
|
- LHCb have flagged up a problem running jobs on the batch nodes being used to test the new worker node configuration. A likely cause has been identified and a fix is being rolled out.
- A configuraion error with the network interface on one of the new batches of worker nodes was identified alst week. These have been drained out and re-installed to fix this. They are expected to go back into production imminently.
- A problem transferring files from FNAL to us is being investigated.
- There was a problem with the test FTS3 service at the end of last week. The underlying cause having been triggered by the problem that affected our network on Tuesday (23rd June) and was reported last week. Atlas flipped transfers to use the production service.
Resolved Disk Server Issues
|
- The draining of GDSS763 (AtlasDataDisk - D1T0) has been complerted and the server removed. This server failed on the 19th June. Following further crashes the disks were placed in a different server/chassis so that all files could be drained off.
Current operational status and issues
|
- We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
- The post mortem review of the network incident on the 8th April is being finalised.
- The intermittent, low-level, load-related packet loss over the OPN to CERN is still being tracked.
Ongoing Disk Server Issues
|
Notable Changes made since the last meeting.
|
- On Wednesday (just after last week's meeting) a network test was carried out. Thi confirmed that non-Tier1 services on our network now route traffic avoiding our (problematic) Tier1 router pair. The test also coinfirmed the long-standing problem in the primary Tier1 router. This test paves the way for a longer intervention, with the vendor present, to try and get to the bottom of the router problem.
- Old files in Castor that were missing stored checksums have had these added.
- Eight additional disk servers have been added to AtlasDataDisk (approaching a Petabyte of extra capacity).
- The batch job limit for Alice has been completely removed. (It was set at 6000).
- A detailed change to a database procedure has been made to the LHCb Castor stager (yesterday - Tuesday 23rd) and then to CMS today (Wed 24th). This change significantly speeds up file open times within Castor.
- Files are being successfull transferred from Durham for Dirac.
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
Whole site
|
SCHEDULED
|
WARNING
|
01/07/2015 10:00
|
01/07/2015 11:30
|
1 hour and 30 minutes
|
Warning on site during quarterly UPS/Generator load test.
|
arc-ce05.gridpp.rl.ac.uk
|
SCHEDULED
|
OUTAGE
|
10/06/2015 12:00
|
08/07/2015 12:00
|
28 days,
|
This particular ARC CE will be decommisioned. (Four other ARC CEs remain in production).
|
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases: Castor, Atlas Frontier (LFC done)
- Castor:
- Update SRMs to new version (includes updating to SL6).
- Update the Oracle databses behind Castor to version 11.2.0.4. Will require some downtimes (See above)
- Update disk servers to SL6.
- Update to Castor version 2.1.15.
- Networking:
- Resolve problems with primary Tier1 Router. Need to schedule a roughly half-day outage for the vendor to carry out investigations.
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
- Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
- Make routing changes to allow the removal of the UKLight Router.
- Cabling/switch changes to the network in the UPS room to improve resilience.
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
Entries in GOC DB starting since the last report.
|
Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
Whole site
|
SCHEDULED
|
WARNING
|
01/07/2015 10:00
|
01/07/2015 11:30
|
1 hour and 30 minutes
|
Warning on site during quarterly UPS/Generator load test.
|
arc-ce05.gridpp.rl.ac.uk
|
SCHEDULED
|
OUTAGE
|
10/06/2015 12:00
|
08/07/2015 12:00
|
28 days,
|
This particular ARC CE will be decommisioned. (Four other ARC CEs remain in production).
|
Open GGUS Tickets (Snapshot during morning of meeting)
|
GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
114512
|
Green
|
Less Urgent
|
In Progress
|
2015-06-12
|
2015-06-22
|
Atlas
|
deletion errors for RAL-LCG2
|
113910
|
Green
|
Less Urgent
|
Waiting Reply
|
2015-05-26
|
2015-06-23
|
SNO+
|
RAL data staging
|
113836
|
Green
|
Less Urgent
|
In Progress
|
2015-05-20
|
2015-06-24
|
|
GLUE 1 vs GLUE 2 mismatch in published queues
|
112721
|
Red
|
Less Urgent
|
In Progress
|
2015-03-28
|
2015-06-23
|
Atlas
|
RAL-LCG2: SOURCE Failed to get source file size
|
109694
|
Red
|
Urgent
|
Waiting Reply
|
2014-11-03
|
2015-06-23
|
SNO+
|
gfal-copy failing for files at RAL
|
108944
|
Red
|
Less Urgent
|
In Progress
|
2014-10-01
|
2015-06-16
|
CMS
|
AAA access test failing at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
24/06/15 |
100 |
100 |
100 |
100 |
100 |
100 |
99 |
|
25/06/15 |
100 |
100 |
100 |
100 |
100 |
97 |
99 |
|
26/06/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
27/06/15 |
100 |
100 |
100 |
100 |
100 |
98 |
n/a |
|
28/06/15 |
100 |
100 |
100 |
100 |
100 |
98 |
100 |
|
29/06/15 |
100 |
100 |
100 |
100 |
100 |
98 |
100 |
|
30/06/15 |
54.9 |
100 |
100 |
100 |
100 |
85 |
100 |
|