RAL Tier1 Operations Report for 14th May 2014
| Review of Issues during the week 7th to 14th May 2014.
|
- There are some ongoing grumbly problems with the WMSs. The developers have been contacted.
- A problem has been uncovered while investigating a missing Atlas file. Some half dozen files were lost during the draining of a disk server at the end of February. We are working to check if this is the only such instance of this and (with the developers) to understand the problem. Investigations so far suggest this is not a widespread problem. Draining of disk servers has been stopped while this investigation takes place.
- Around ten days ago a GGUS ticket was escalated to an 'alarm' ticket by LHCb but the Tier1 pager failed to call out. This has been traced to a a problem with the e-mail from GGUS. The GGUS team were contacted. They have fixed the problem and we have tested that the alarm tickets now work again.
| Resolved Disk Server Issues
|
| Current operational status and issues
|
| Ongoing Disk Server Issues
|
| Notable Changes made this last week.
|
- Testing CVMFS Client version 2.1.19 ongoing. This is now rolled out to one batch of worker nodes. So far so good.
- pheno & enmr.eu added to list of authorized VOs on ARC CEs
- FTS3 updated to version 3.2.22 this morning.
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| srm-lhcb-tape.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
20/05/2014 08:00
|
20/05/2014 11:00
|
3 hours
|
Outage of tape system for update of tape library controller. (Postponed from 13th May).
|
| All SRMs
|
SCHEDULED
|
WARNING
|
20/05/2014 08:00
|
20/05/2014 11:00
|
3 hours
|
Outage of tape system for update of tape library controller. (Postponed from 13th May).
|
| lcgvo08.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
15/05/2014 15:00
|
15/05/2014 16:00
|
1 hour
|
Downtime for system maintenance
|
| lcgvo07.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
15/05/2014 14:00
|
15/05/2014 15:00
|
1 hour
|
Downtime for system maintenance
|
| lcglb02.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
15/05/2014 13:00
|
15/05/2014 14:00
|
1 hour
|
Downtime for system maintenance
|
| lcglb01.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
15/05/2014 10:45
|
15/05/2014 11:45
|
1 hour
|
Downtime for system maintenance
|
| lcglb04.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
15/05/2014 09:30
|
15/05/2014 10:30
|
1 hour
|
Downtime for system maintenance
|
| lcgui02.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
30/04/2014 14:00
|
29/05/2014 13:00
|
28 days, 23 hours
|
Service being decommissioned.
|
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
- Provisional dates for the Castor 2.1.14 upgrade delayed to: Nameserver: Tuesday 10th June; Stagers to follow.
- We are starting to plan the termination of the FTS2 service now that almost all use is on FTS3.
Listing by category:
- Databases:
- Switch LFC/FTS/3D to new Database Infrastructure.
- Castor:
- Castor 2.1.14 testing was largely complete, although a new minor version (2.1.14-12) will be released soon.
- Networking:
- Move switches connecting recent disk servers batches ('11, '12) onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Fabric
- We are phasing out the use of the software server used by the small VOs.
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.
| Entries in GOC DB starting between the 7th and 14th May 2014.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| lcgwms06.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
14/05/2014 14:00
|
14/05/2014 16:00
|
2 hours
|
Downtime for system maintenance
|
| lcgwms05.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
14/05/2014 11:30
|
14/05/2014 13:30
|
2 hours
|
Downtime for system maintenance
|
| lcgfts3.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
14/05/2014 10:00
|
14/05/2014 12:00
|
2 hours
|
FTS3 service at RAL unavailable for update to version 3.2.22
|
| lcgwms04.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
14/05/2014 09:30
|
14/05/2014 10:46
|
1 hour and 16 minutes
|
Downtime for system maintenance
|
| lcgui02.gridpp.rl.ac.uk,
|
SCHEDULED
|
OUTAGE
|
30/04/2014 14:00
|
29/05/2014 13:00
|
28 days, 23 hours
|
Service being decommissioned.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 105374
|
Green
|
Less Urgent
|
In Progress
|
2014-05-13
|
2014-05-14
|
CMS
|
RAL: migration to tape
|
| 105308
|
Green
|
Less Urgent
|
In Progress
|
2014-05-11
|
2014-05-13
|
Atlas
|
Jobs at RAL-LCG2_MCORE are failing with "Failed to open shared memory object: Permission denied"
|
| 105161
|
Green
|
Less Urgent
|
In Progress
|
2014-05-05
|
2014-05-13
|
H1
|
hone jobs submitted into CREAM queues through lcgwms05.gridpp.rl.ac.uk & lcgwms06.gridpp.rl.ac.uk WMSs are are Ready status long time (more as 5 hours)
|
| 105100
|
Green
|
Urgent
|
On Hold
|
2014-05-02
|
2014-05-12
|
CMS
|
T1_UK_RAL Consistency Check (May14)
|
| 98249
|
Red
|
Urgent
|
Waiting Reply
|
2013-10-21
|
2014-05-12
|
SNO+
|
please configure cvmfs stratum-0 for SNO+ at RAL T1
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
CMS HC |
Comment
|
| 07/05/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 08/05/14 |
100 |
100 |
100 |
100 |
100 |
94 |
n/a |
|
| 09/05/14 |
100 |
100 |
100 |
100 |
100 |
100 |
n/a |
|
| 10/05/14 |
100 |
100 |
100 |
100 |
100 |
100 |
n/a |
|
| 11/05/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 12/05/14 |
100 |
100 |
100 |
100 |
100 |
100 |
100 |
|
| 13/05/14 |
100 |
100 |
100 |
100 |
100 |
100 |
99 |
|
