RAL Tier1 Operations Report for 8th March 2017
| Review of Issues during the week 1st to 8th March 2017.
|
- Following a discussion at last week's liaison meeting an apparant cap on LHCb batch jobs was removed. However, it turned out that the restriction in the Condor configuration file was not erroneous and had no effect. LHCb batch jobs were not being limited.
- There was a problem reported for access to AtlasScratchDisk in Castor this morning (8th Mar). Atlas have reported a large backlog of outstanding file transfers. Being worked on at time of meeting.
| Resolved Disk Server Issues
|
| Current operational status and issues
|
- We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities but the level of failures is reduced as compared to a few weeks ago.
| Ongoing Disk Server Issues
|
- GDSS689 (AtlasDataDisk - D1T0) reported 'fsprobe' errors and was taken out of production this morning. Investigations ongoing.
| Limits on concurrent batch system jobs.
|
- Atlas Pilot (Analysis) 1500
- CMS Multicore 460
| Notable Changes made since the last meeting.
|
- CMS PhEDEx debug transfers switched from CASTOR to CEPH ECHO.
- Ongoing work applying security and other patches. More back-end database systems have been updated to remove a software layer ("asmlib").
- IPv6 has been disabled across systems in preparation for enabling IPv6 through the routers.
- This morning (8th March) work has been carried out to enabled IPv6 through the Tier's routers.
None
| Advanced warning for other interventions
|
| The following items are being discussed and are still to be formally scheduled and announced.
|
Pending - but not yet formally announced:
- Update Castor SRMs. Propose LHCb SRMs first - target date 22nd March.
- Chiller replacement - work imminent.
- Merge AtlasScratchDisk into larger Atlas disk pool.
Listing by category:
- Castor:
- Update SRMs to new version, including updating to SL6.
- Bring some newer disk servers ('14 generation) into service, replacing some older ('12 generation) servers.
- Databases
- Removal of "asmlib" layer on Oracle database nodes. (Ongoing)
- Infrastructure:
- Two of the chillers supplying the air-conditioning for the R89 machine room will be replaced.
| Entries in GOC DB starting since the last report.
|
| Service
|
Scheduled?
|
Outage/At Risk
|
Start
|
End
|
Duration
|
Reason
|
| Whole site
|
SCHEDULED
|
WARNING
|
08/03/2017 07:00
|
08/03/2017 11:00
|
4 hours
|
Warning on site during network intervention in preparation for IPv6.
|
| Open GGUS Tickets (Snapshot during morning of meeting)
|
| GGUS ID |
Level |
Urgency |
State |
Creation |
Last Update |
VO |
Subject
|
| 126905
|
Green
|
Less Urgent
|
In Progress
|
2017-03-02
|
2017-03-02
|
solid
|
finish commissioning cvmfs server for solidexperiment.org
|
| 126184
|
Green
|
Less Urgent
|
In Progress
|
2017-01-26
|
2017-02-07
|
Atlas
|
Request of inputs for new sites monitoring
|
| 124876
|
Red
|
Less Urgent
|
On Hold
|
2016-11-07
|
2017-01-01
|
OPS
|
[Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
|
| 117683
|
Red
|
Less Urgent
|
On Hold
|
2015-11-18
|
2017-03-02
|
|
CASTOR at RAL not publishing GLUE 2.
|
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC ECHO = Atlas ECHO (Template 842);CMS HC = CMS HammerCloud
| Day |
OPS |
Alice |
Atlas |
CMS |
LHCb |
Atlas HC |
Atlas HC ECHO |
CMS HC |
Comment
|
| 01/03/17 |
100 |
100 |
83 |
81 |
100 |
99 |
100 |
99 |
Atlas: Ongoing problems with SRM test; CMS - CE test failures due to poblem with argus server.
|
| 02/03/17 |
100 |
100 |
59 |
99 |
100 |
100 |
99 |
100 |
Atlas: Ongoing problems with SRM test; CMS - timeouts in SRM tests.
|
| 03/03/17 |
100 |
100 |
91 |
98 |
100 |
97 |
100 |
100 |
Atlas: Ongoing problems with SRM test; CMS - timeouts in SRM tests.
|
| 04/03/17 |
100 |
100 |
96 |
100 |
100 |
100 |
100 |
99 |
Atlas: Ongoing problems with SRM test.
|
| 05/03/17 |
100 |
97 |
98 |
88 |
92 |
97 |
100 |
100 |
Atlas: Ongoing problems with SRM test; CMS - timeouts in SRM tests; LHCb - some SRM test failures.
|
| 06/03/17 |
100 |
68 |
89 |
97 |
100 |
99 |
98 |
100 |
Alice: Central monitoring problem; Atlas: Ongoing problems with SRM test; CMS - timeouts in SRM tests.
|
| 07/03/17 |
100 |
100 |
100 |
93 |
100 |
94 |
99 |
100 |
Timeouts on CMS SRM tests.
|
- ECHO: Two additional 'MON' boxes are being set-up bringing the total to five. The existing three can cope with normal activity but the additional ones would speed up recoveries and starts. Two additional gateway nodes are also being set-up (also bringing the total to five) which will improve bandwidth.
- VO Dirac: some tar files moved from Edinburgh but little activity from sites other than Durham as yet.
