Difference between revisions of "Tier1 Operations Report 2018-01-17"
From GridPP Wiki
(→) |
(→) |
||
(11 intermediate revisions by 2 users not shown) | |||
Line 9: | Line 9: | ||
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | {| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | ||
|- | |- | ||
− | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week | + | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 10th January 2018 to 17th January 2018 |
|} | |} | ||
− | + | * Over the weekend (13/14 Jan) LHCb were running merging jobs and this caused problems for LHCb Castor. The on-call team responded - and the problems stopped when LHCb eased back on this workload. | |
+ | * Also over the weekend there was a problem with the Atlas Castor SRMs which stopped working for some hours. The SRMs started working on their own. However, the problem has recurred and further investigations have been taking place. | ||
+ | * CMS AAA redirection problems were reported at the end of last week. Investigations continue but this appears to be affecting other sites too and may not be a local RAL Tier1 problem. | ||
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
<!-- *********************************************************** -----> | <!-- *********************************************************** -----> | ||
Line 33: | Line 35: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues | ||
|} | |} | ||
− | * gdss736 ( | + | * gdss736 (LHCbDst - D1T0) – rebuilt and back in production (RO) |
− | * | + | * gdss728 (CMSDisk - D1T0) – rebuilt and back in production (RW) |
<!-- ***************************************************** -----> | <!-- ***************************************************** -----> | ||
Line 44: | Line 46: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | ||
|} | |} | ||
− | * | + | * gdss717 (CMSTape - D0T1) – multiple drive failure |
<!-- ***************End Ongoing Disk Server Issues**************** -----> | <!-- ***************End Ongoing Disk Server Issues**************** -----> | ||
<!-- ************************************************************* -----> | <!-- ************************************************************* -----> | ||
Line 66: | Line 68: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting. | ||
|} | |} | ||
− | * | + | * Security patching done or underway. |
− | + | ||
<!-- *************End Notable Changes made this last week************** -----> | <!-- *************End Notable Changes made this last week************** -----> | ||
<!-- ****************************************************************** -----> | <!-- ****************************************************************** -----> | ||
Line 78: | Line 79: | ||
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report. | | style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report. | ||
|} | |} | ||
− | + | * None | |
<!-- **********************End GOC DB Entries************************** -----> | <!-- **********************End GOC DB Entries************************** -----> | ||
<!-- ****************************************************************** -----> | <!-- ****************************************************************** -----> | ||
Line 99: | Line 100: | ||
! Reason | ! Reason | ||
|- | |- | ||
− | | | + | |srm-alice.gridpp.rl.ac.uk, srm-atlas.gridpp.rl.ac.uk, srm-biomed.gridpp.rl.ac.uk, srm-cert.gridpp.rl.ac.uk, srm-cms-disk.gridpp.rl.ac.uk, srm-cms.gridpp.rl.ac.uk, srm-dteam.gridpp.rl.ac.uk, srm-ilc.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-mice.gridpp.rl.ac.uk, srm-minos.gridpp.rl.ac.uk, srm-na62.gridpp.rl.ac.uk, srm-pheno.gridpp.rl.ac.uk, srm-preprod.gridpp.rl.ac.uk, srm-snoplus.gridpp.rl.ac.uk, srm-solid.gridpp.rl.ac.uk, srm-t2k.gridpp.rl.ac.uk, |
+ | | SCHEDULED | ||
+ | | OUTAGE | ||
+ | | 17/01/2018 10:00 | ||
+ | | 17/01/2018 13:00 | ||
+ | | 3 hours | ||
+ | |Outage of Castor Storage to apply Security patches. | ||
+ | |- | ||
+ | |lcglb01.gridpp.rl.ac.uk, lcglb02.gridpp.rl.ac.uk, lcgwms04.gridpp.rl.ac.uk, lcgwms05.gridpp.rl.ac.uk, | ||
| SCHEDULED | | SCHEDULED | ||
| OUTAGE | | OUTAGE | ||
Line 105: | Line 114: | ||
| 19/01/2018 12:00 | | 19/01/2018 12:00 | ||
| 7 days, 2 hours | | 7 days, 2 hours | ||
− | | WMS Decommissioning RAL Tier1 | + | |WMS Decommissioning RAL Tier1 |
|} | |} | ||
<!-- **********************End GOC DB Entries************************** -----> | <!-- **********************End GOC DB Entries************************** -----> | ||
Line 131: | Line 140: | ||
** Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers). | ** Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers). | ||
* Services | * Services | ||
+ | ** Ending of WMS service (ongoing). | ||
* Internal | * Internal | ||
** DNS servers will be rolled out within the Tier1 network. | ** DNS servers will be rolled out within the Tier1 network. | ||
Line 159: | Line 169: | ||
! Subject | ! Subject | ||
|- | |- | ||
− | | style="background-color: green;" | | + | | style="background-color: green;" | 132844 |
+ | | TEAM | ||
+ | | atlas | ||
+ | | RAL-LCG2 | ||
+ | | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | ||
+ | | waiting for reply | ||
+ | | urgent | ||
+ | | 2018-01-14 02:10:00 | ||
+ | | 2018-01-16 16:51:00 | ||
+ | | Storage Systems | ||
+ | | UK RAL-LCG2 DATADISK transfer errors "DESTINATION OVERWRITE srm-ifce err:" | ||
+ | |- | ||
+ | | style="background-color: green;" | 132830 | ||
| USER | | USER | ||
− | | | + | | cms |
| RAL-LCG2 | | RAL-LCG2 | ||
| NGI_UK | | NGI_UK | ||
| in progress | | in progress | ||
− | | | + | | very urgent |
− | | 2018-01- | + | | 2018-01-12 11:11:00 |
− | | 2018-01- | + | | 2018-01-16 16:36:00 |
− | | | + | | CMS_AAA WAN Access |
− | | | + | | Reading issues T1_UK_RAL |
+ | |- | ||
+ | | style="background-color: green;" | 132802 | ||
+ | | USER | ||
+ | | cms | ||
+ | | RAL-LCG2 | ||
+ | | NGI_UK | ||
+ | | in progress | ||
+ | | urgent | ||
+ | | 2018-01-11 10:30:00 | ||
+ | | 2018-01-15 11:26:00 | ||
+ | | CMS_AAA WAN Access | ||
+ | | Low HC xrootd success rates at T1_UK_RAL | ||
|- | |- | ||
| style="background-color: green;" | 132712 | | style="background-color: green;" | 132712 | ||
Line 175: | Line 209: | ||
| other | | other | ||
| RAL-LCG2 | | RAL-LCG2 | ||
− | | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | + | | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:Kashif.Mohammad@PHYSICS.OX.AC.UK |
| in progress | | in progress | ||
| less urgent | | less urgent | ||
| 2018-01-04 16:22:00 | | 2018-01-04 16:22:00 | ||
− | | 2018-01- | + | | 2018-01-16 15:01:00 |
| Other | | Other | ||
| support for the hyperk VO (RAL-LCG2) | | support for the hyperk VO (RAL-LCG2) | ||
|- | |- | ||
− | | style="background-color: | + | | style="background-color: yellow;" | 132589 |
| TEAM | | TEAM | ||
| lhcb | | lhcb | ||
Line 191: | Line 225: | ||
| very urgent | | very urgent | ||
| 2017-12-21 06:45:00 | | 2017-12-21 06:45:00 | ||
− | | 2018-01- | + | | 2018-01-10 11:07:00 |
| Local Batch System | | Local Batch System | ||
| Killed pilots at RAL | | Killed pilots at RAL | ||
|- | |- | ||
− | | style="background-color: | + | | style="background-color: orange;" | 131815 |
| USER | | USER | ||
| t2k.org | | t2k.org | ||
| RAL-LCG2 | | RAL-LCG2 | ||
| NGI_UK | | NGI_UK | ||
− | | | + | | waiting for reply |
| less urgent | | less urgent | ||
| 2017-11-13 14:42:00 | | 2017-11-13 14:42:00 | ||
− | | | + | | 2018-01-17 09:24:00 |
| Storage Systems | | Storage Systems | ||
| Extremely long download times for T2K files on tape at RAL | | Extremely long download times for T2K files on tape at RAL | ||
Line 260: | Line 294: | ||
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas Echo !! Comment | ! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas Echo !! Comment | ||
|- | |- | ||
− | | | + | | 10/01/18 || 100 || 100 || 100 || 100 || 100 || 100 || |
|- | |- | ||
− | | | + | | 11/01/18 || 100 || 100 || 100 || 100 || style="background-color: yellow;" |98|| 100 || gdss736 (LHCb) failed |
|- | |- | ||
− | | | + | | 12/01/18 || 100 || 100 || 100 || 100 || 100|| 100 || |
|- | |- | ||
− | | | + | | 13/01/18 || 100 || 100 || 100 || 100 || 100 || 100 || |
|- | |- | ||
− | | | + | | 14/01/18 || 100 || style="background-color: yellow;" | 96 || style="background-color: yellow;" | 53 || 100 || 100 || 100 || |
|- | |- | ||
− | | | + | | 15/01/18 || 100 || 100 || style="background-color: yellow;" |74|| 100 || 100 || 100 || |
|- | |- | ||
− | | | + | | 16/01/18 || 100 || 100 || style="background-color: yellow;" |92|| 100 || 100 || 100 || |
|} | |} | ||
Line 288: | Line 322: | ||
! Day !! Atlas HC !! CMS HC !! Comment | ! Day !! Atlas HC !! CMS HC !! Comment | ||
|- | |- | ||
− | | | + | | 10/01/18 || 100 || 100 || |
|- | |- | ||
− | | | + | | 11/01/18 || 100 || 100 || |
|- | |- | ||
− | | | + | | 12/01/18 || 100 || 100 || |
|- | |- | ||
− | | | + | | 13/01/18 || 100 || 100 || |
|- | |- | ||
− | | | + | | 14/01/18 || 100 || 100 || |
|- | |- | ||
− | | | + | | 15/01/18 || 100 || style="background-color: yellow;" | 99 || |
|- | |- | ||
− | | | + | | 16/01/18 || style="background-color: yellow;" | 85 || style="background-color: yellow;" | 99 || |
|} | |} | ||
Latest revision as of 13:39, 17 January 2018
RAL Tier1 Operations Report for 17th January 2018
Review of Issues during the week 10th January 2018 to 17th January 2018 |
- Over the weekend (13/14 Jan) LHCb were running merging jobs and this caused problems for LHCb Castor. The on-call team responded - and the problems stopped when LHCb eased back on this workload.
- Also over the weekend there was a problem with the Atlas Castor SRMs which stopped working for some hours. The SRMs started working on their own. However, the problem has recurred and further investigations have been taking place.
- CMS AAA redirection problems were reported at the end of last week. Investigations continue but this appears to be affecting other sites too and may not be a local RAL Tier1 problem.
Current operational status and issues |
- Ongoing security patching.
Resolved Castor Disk Server Issues |
- gdss736 (LHCbDst - D1T0) – rebuilt and back in production (RO)
- gdss728 (CMSDisk - D1T0) – rebuilt and back in production (RW)
Ongoing Castor Disk Server Issues |
- gdss717 (CMSTape - D0T1) – multiple drive failure
Limits on concurrent batch system jobs. |
- CMS Multicore 550
Notable Changes made since the last meeting. |
- Security patching done or underway.
Entries in GOC DB starting since the last report. |
- None
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
srm-alice.gridpp.rl.ac.uk, srm-atlas.gridpp.rl.ac.uk, srm-biomed.gridpp.rl.ac.uk, srm-cert.gridpp.rl.ac.uk, srm-cms-disk.gridpp.rl.ac.uk, srm-cms.gridpp.rl.ac.uk, srm-dteam.gridpp.rl.ac.uk, srm-ilc.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-lhcb.gridpp.rl.ac.uk, srm-mice.gridpp.rl.ac.uk, srm-minos.gridpp.rl.ac.uk, srm-na62.gridpp.rl.ac.uk, srm-pheno.gridpp.rl.ac.uk, srm-preprod.gridpp.rl.ac.uk, srm-snoplus.gridpp.rl.ac.uk, srm-solid.gridpp.rl.ac.uk, srm-t2k.gridpp.rl.ac.uk, | SCHEDULED | OUTAGE | 17/01/2018 10:00 | 17/01/2018 13:00 | 3 hours | Outage of Castor Storage to apply Security patches. |
lcglb01.gridpp.rl.ac.uk, lcglb02.gridpp.rl.ac.uk, lcgwms04.gridpp.rl.ac.uk, lcgwms05.gridpp.rl.ac.uk, | SCHEDULED | OUTAGE | 12/01/2018 10:00 | 19/01/2018 12:00 | 7 days, 2 hours | WMS Decommissioning RAL Tier1 |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Ongoing or Pending - but not yet formally announced:
Listing by category:
- Castor:
- Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
- Move to generic Castor headnodes.
- Echo:
- Update to next CEPH version ("Luminous").
- Networking
- Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
- Services
- Ending of WMS service (ongoing).
- Internal
- DNS servers will be rolled out within the Tier1 network.
- Infrastructure
- Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.
Open GGUS Tickets (Snapshot during morning of meeting) |
Ticket-ID | Type | VO | Notified Site | Resp. Unit | Status | Priority | Creation | Last Update | ToI | Subject |
---|---|---|---|---|---|---|---|---|---|---|
132844 | TEAM | atlas | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | waiting for reply | urgent | 2018-01-14 02:10:00 | 2018-01-16 16:51:00 | Storage Systems | UK RAL-LCG2 DATADISK transfer errors "DESTINATION OVERWRITE srm-ifce err:" |
132830 | USER | cms | RAL-LCG2 | NGI_UK | in progress | very urgent | 2018-01-12 11:11:00 | 2018-01-16 16:36:00 | CMS_AAA WAN Access | Reading issues T1_UK_RAL |
132802 | USER | cms | RAL-LCG2 | NGI_UK | in progress | urgent | 2018-01-11 10:30:00 | 2018-01-15 11:26:00 | CMS_AAA WAN Access | Low HC xrootd success rates at T1_UK_RAL |
132712 | USER | other | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:Kashif.Mohammad@PHYSICS.OX.AC.UK | in progress | less urgent | 2018-01-04 16:22:00 | 2018-01-16 15:01:00 | Other | support for the hyperk VO (RAL-LCG2) |
132589 | TEAM | lhcb | RAL-LCG2 | NGI_UK | in progress | very urgent | 2017-12-21 06:45:00 | 2018-01-10 11:07:00 | Local Batch System | Killed pilots at RAL |
131815 | USER | t2k.org | RAL-LCG2 | NGI_UK | waiting for reply | less urgent | 2017-11-13 14:42:00 | 2018-01-17 09:24:00 | Storage Systems | Extremely long download times for T2K files on tape at RAL |
127597 | USER | cms | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:sexton@fnal.gov | on hold | urgent | 2017-04-07 10:34:00 | 2017-10-05 09:14:00 | File Transfer | Check networking and xrootd RAL-CERN performance |
124876 | USER | ops | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | on hold | less urgent | 2016-11-07 12:06:00 | 2017-11-13 16:55:00 | Operations | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk |
117683 | USER | none | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | on hold | less urgent | 2015-11-18 11:36:00 | 2018-01-03 15:26:00 | Information System | CASTOR at RAL not publishing GLUE 2 |
Availability Report |
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas Echo | Comment |
---|---|---|---|---|---|---|---|
10/01/18 | 100 | 100 | 100 | 100 | 100 | 100 | |
11/01/18 | 100 | 100 | 100 | 100 | 98 | 100 | gdss736 (LHCb) failed |
12/01/18 | 100 | 100 | 100 | 100 | 100 | 100 | |
13/01/18 | 100 | 100 | 100 | 100 | 100 | 100 | |
14/01/18 | 100 | 96 | 53 | 100 | 100 | 100 | |
15/01/18 | 100 | 100 | 74 | 100 | 100 | 100 | |
16/01/18 | 100 | 100 | 92 | 100 | 100 | 100 |
Hammercloud Test Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_UCORE, Template 841); CMS HC = CMS HammerCloud
Day | Atlas HC | CMS HC | Comment |
---|---|---|---|
10/01/18 | 100 | 100 | |
11/01/18 | 100 | 100 | |
12/01/18 | 100 | 100 | |
13/01/18 | 100 | 100 | |
14/01/18 | 100 | 100 | |
15/01/18 | 100 | 99 | |
16/01/18 | 85 | 99 |
Notes from Meeting. |
- None yet.