RAL Tier1 Operations Report for 10th January 2018
Review of Issues during the week 3rd January 2018 to 10th January 2018
|
Current operational status and issues
|
- gdss745 (Atlas) - server to be considered 100% loss.
Resolved Castor Disk Server Issues
|
- gdss736 (LHCb)
- gdss756 (CMS)
Ongoing Castor Disk Server Issues
|
Limits on concurrent batch system jobs.
|
Notable Changes made since the last meeting.
|
- The ATLAS quota on Echo was increased by 500TB yesterday (2nd Jan). It now stands at 4.6PBytes.
- A round of Linux kernel patching of back-end database systems has been completed.
Entries in GOC DB starting since the last report.
|
No downtime scheduled in the GOCDB between 2017-12-20 and 2018-01-03
Advanced warning for other interventions
|
The following items are being discussed and are still to be formally scheduled and announced.
|
Ongoing or Pending - but not yet formally announced:
Listing by category:
- Castor:
- Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
- Move to generic Castor headnodes.
- Echo:
- Update to next CEPH version ("Luminous").
- Networking
- Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
- Services
- Internal
- DNS servers will be rolled out within the Tier1 network.
- Infrastructure
- Testing of power distribution boards in the R89 machine room is being scheduled for some time in March. The effect of this on our services is being discussed.
Open GGUS Tickets (Snapshot during morning of meeting)
|
Ticket-ID
|
Type
|
VO
|
Notified Site
|
Resp. Unit
|
Status
|
Priority
|
Creation
|
Last Update
|
ToI
|
Subject
|
132748
|
USER
|
ops
|
RAL-LCG2
|
NGI_UK
|
in progress
|
less urgent
|
2018-01-08 14:45:00
|
2018-01-09 09:58:00
|
Operations
|
[Rod Dashboard] Issues detected at RAL-LCG2
|
132712
|
USER
|
other
|
RAL-LCG2
|
NGI_UK assign to:lcg-support@gridpp.rl.ac.uk
|
in progress
|
less urgent
|
2018-01-04 16:22:00
|
2018-01-08 13:35:00
|
Other
|
support for the hyperk VO (RAL-LCG2)
|
132589
|
TEAM
|
lhcb
|
RAL-LCG2
|
NGI_UK
|
in progress
|
very urgent
|
2017-12-21 06:45:00
|
2018-01-03 15:21:00
|
Local Batch System
|
Killed pilots at RAL
|
131815
|
USER
|
t2k.org
|
RAL-LCG2
|
NGI_UK
|
in progress
|
less urgent
|
2017-11-13 14:42:00
|
2017-12-01 19:30:00
|
Storage Systems
|
Extremely long download times for T2K files on tape at RAL
|
127597
|
USER
|
cms
|
RAL-LCG2
|
NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:sexton@fnal.gov
|
on hold
|
urgent
|
2017-04-07 10:34:00
|
2017-10-05 09:14:00
|
File Transfer
|
Check networking and xrootd RAL-CERN performance
|
124876
|
USER
|
ops
|
RAL-LCG2
|
NGI_UK assign to:lcg-support@gridpp.rl.ac.uk
|
on hold
|
less urgent
|
2016-11-07 12:06:00
|
2017-11-13 16:55:00
|
Operations
|
[Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
|
117683
|
USER
|
none
|
RAL-LCG2
|
NGI_UK assign to:lcg-support@gridpp.rl.ac.uk
|
on hold
|
less urgent
|
2015-11-18 11:36:00
|
2018-01-03 15:26:00
|
Information System
|
CASTOR at RAL not publishing GLUE 2
|
Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas Echo |
Comment
|
03/01/18 |
100 |
100 |
100 |
100 |
100 |
100 |
|
04/01/18 |
100 |
100 |
100 |
100 |
100 |
100 |
|
05/01/18 |
100 |
100 |
100 |
100 |
100 |
100 |
|
06/01/18 |
100 |
100 |
100 |
100 |
100 |
100 |
|
07/01/18 |
100 |
100 |
100 |
100 |
100 |
100 |
|
08/01/18 |
100 |
100 |
100 |
100 |
100 |
100 |
|
09/01/18 |
100 |
100 |
100 |
100 |
100 |
100 |
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC Echo = Atlas Echo (Template 841);CMS HC = CMS HammerCloud
Day |
Atlas HC |
CMS HC |
Comment
|
03/01/18 |
100 |
100 |
100
|
04/01/18 |
100 |
100 |
100
|
05/01/18 |
100 |
100 |
100
|
06/01/18 |
100 |
100 |
100
|
07/01/18 |
100 |
100 |
100
|
08/01/18 |
100 |
100 |
100
|
09/01/18 |
100 |
100 |
100
|
- CMS quota on Echo will be increased from 250 to 500TB. CMS will then do more stress testing.
- MICE have probably taken their final data. Some problems with rate of transfer of data into Castor will no longer be relevant.