RAL Tier1 Operations Report for 24th June 2015
| Review of Issues during the week 17th to 24th June 2015.
|
- The behaviour of the CMS Castor instance has been improved since a change made last week.
- There has been a problem with job submission for MICE ("no compatible resources" from the BDII) reported on 10th June.
- A problem with a couple of entries in the Castor gridmap file was flagged up by LHCb. This was chased to the relevant updating script pointing at an old VOMS server and fixed this morning (Wed 17th).
- AtlasDataDisk in Castor became full this morning (Wed 17th June). Additional disk servers are in the process of being deployed to this service class.
- LHCb have flagged up a problem reading very old files that pre-date the checksums being enabled in Castor. Some of these have had their checksum added - but there are more to be done.
| Resolved Disk Server Issues
|
| Current operational status and issues
|
- We are running with a single router connecting the Tier1 network to the site network, rather than a resilient pair.
- Castor xroot performance problems seen by CMS has been improved. The proof of this awaits further running of problematic job mixes by CMS but looks promising.
- The post mortem review of the network incident on the 8th April is being finalised.
- The intermittent, low-level, load-related packet loss over the OPN to CERN is still being tracked.
| Ongoing Disk Server Issues
|
| Notable Changes made since the last meeting.
|
- On Wednesday (10th June) access to arc-ce05 was stopped pending its full decommissioning.
- On Wednedsay (10th June) the CMS Castor instance was modified to allow unscheduled reads when using xroot. Performance so far suggests a big improvement.
- Urgent updates to version 2.9.3-1 were applied to both test and production FTS3 systems on Thursday (11th June).
- On Monday (15th June) the Castor rebalancer was enabled across all instances.
- Access to the LFC has been enabled for the DiRAC VO.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| arc-ce05.gridpp.rl.ac.uk
|
SCHEDULED
|
OUTAGE
|
10/06/2015 12:00
|
08/07/2015 12:00
|
28 days,
|
This particular ARC CE will be decommisioned. (Four other ARC CEs remain in production).
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- A short (fifteen minute) test needs to take place to confirm we have successfully separated the non-Tier1 traffic off the router pair. This will require a break in connectivity to the Tier1. (We will also make use of this short outage to stop a small number of additional services while the hypervisor hosting them is physically moved.) Once this test has been done a longer (few hour) intervention can be scheduled to investigate the router problems.
Listing by category:
- Databases:
- Switch LFC/3D to new Database Infrastructure.
- Update to Oracle 11.2.0.4. This will affect all services that use Oracle databases: Castor, Atlas Frontier (LFC done)
- Castor:
- Enable disk server rebalancing.
- Update SRMs to new version (includes updating to SL6).
- Update the Oracle databses behind Castor to version 11.2.0.4. Will require some downtimes (See above)
- Update disk servers to SL6.
- Update to Castor version 2.1.15.
- Networking:
- Resolve problems with primary Tier1 Router
- Enable the RIP protocol for updating routing tables on the Tier1 routers. (Install patch to Router software).
- Increase bandwidth of the link from the Tier1 into the RAL internal site network to 40Gbit.
- Make routing changes to allow the removal of the UKLight Router.
- Cabling/switch changes to the network in the UPS room to improve resilience.
- Fabric
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
| Entries in GOC DB starting since the last report.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| Whole site
|
SCHEDULED
|
OUTAGE
|
17/06/2015 13:45
|
17/06/2015 15:15
|
1 hour and 30 minutes
|
During this time window there will be a 15 minute disconnection of the RAL-LCG2 site from the network. This will take place sometime between 13:00 - 13:30 UTC. For this 15-minute period all services will be unavailable. The Castor storage system will be stopped at 12:45 UTC before the network break, and restarted once the 15-minute break is over. The declared time window allows time for Castor and other services to be checked out after the network break. The network outage is for a test of a revised network configuration.
|
| srm-atlas.gridpp.rl.ac.uk,
|
UNSCHEDULED
|
WARNING
|
17/06/2015 10:00
|
17/06/2015 17:00
|
7 hours
|
AtlasDataDisk full. Working to resolve this. Other Atlas areas, and reads from AtlasDataDisk are OK.
|
| arc-ce05.gridpp.rl.ac.uk
|
SCHEDULED
|
OUTAGE
|
10/06/2015 12:00
|
08/07/2015 12:00
|
28 days,
|
This particular ARC CE will be decommisioned. (Four other ARC CEs remain in production).
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 114391
|
Green
|
Urgent
|
In Progress
|
2015-06-17
|
2015-06-17
|
Atlas
|
RAL-LCG2: DESTINATION transfer failure with "disk pool is full" error
|
| 114296
|
Green
|
Top Priority
|
Waiting Reply
|
2015-06-12
|
2015-06-12
|
LHCb
|
Checksum missing for RAL disk resident file replicas
|
| 113910
|
Green
|
Less Urgent
|
In Progress
|
2015-05-26
|
2015-05-28
|
SNO+
|
RAL data staging
|
| 113836
|
Green
|
Less Urgent
|
In Progress
|
2015-05-20
|
2015-06-08
|
|
GLUE 1 vs GLUE 2 mismatch in published queues
|
| 112721
|
Yellow
|
Less Urgent
|
In Progress
|
2015-03-28
|
2015-06-08
|
Atlas
|
RAL-LCG2: SOURCE Failed to get source file size
|
| 109694
|
Red
|
Urgent
|
In Progress
|
2014-11-03
|
2015-05-27
|
SNO+
|
gfal-copy failing for files at RAL
|
| 108944
|
Red
|
Less Urgent
|
In Progress
|
2014-10-01
|
2015-06-16
|
CMS
|
AAA access test failing at T1_UK_RAL
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 17/06/15 |
93.8 |
94.0 |
50.0 |
93.0 |
96.0 |
96 |
98 |
Atlas: DataDiskFull; All: Tests failed during network outage (planned test).
|
| 18/06/15 |
100 |
100 |
100 |
100 |
100 |
97 |
95 |
|
| 19/06/15 |
100 |
100 |
98.0 |
100 |
100 |
93 |
98 |
Single SRM Test failure: Could not open connection to srm-atlas
|
| 20/06/15 |
100 |
100 |
98.0 |
100 |
100 |
100 |
100 |
Single SRM Test failure: Error trying to locate the file in the disk cache
|
| 21/06/15 |
100 |
100 |
100 |
100 |
100 |
96 |
99 |
|
| 22/06/15 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 23/06/15 |
100 |
100 |
100 |
95.0 |
100 |
87 |
99 |
SRM and CE test failures during local network problem.
|
