Difference between revisions of "Tier1 Operations Report 2018-01-03"
From GridPP Wiki
(→) |
|||
(20 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
− | ==RAL Tier1 Operations Report for | + | ==RAL Tier1 Operations Report for 3rd January 2018== |
__NOTOC__ | __NOTOC__ | ||
Line 11: | Line 11: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 21st December 2017 to 3rd January 2018 | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 21st December 2017 to 3rd January 2018 | ||
|} | |} | ||
− | + | * Network connectivity issues on Stack 9 in the UPS room overnight Thursday/Friday 21/22 December affected some internal systems (mainly monitoring). A member of staff attended overnight and a faulty transceiver was found the be the cause and replaced. External services were unaffected. | |
− | + | * Operations over the Christmas and New year holiday were generally stable although not completely quiet for the oncall team. There were some Castor disk server failures and staff did attend site over the holiday to replace failed disk drives. Our monitoring flagged problems with LHCb SAM tests - although these turned out to be a caused by a problem with a LHCb certificate. | |
− | + | * The number of Atlas batch jobs being run is lower than expected. The batch (Condor) scheduling will be looked at to try and understand and improve this. | |
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
<!-- *********************************************************** -----> | <!-- *********************************************************** -----> | ||
Line 37: | Line 37: | ||
* GDSS688 (cmsDisk - D1T0) is back in production. | * GDSS688 (cmsDisk - D1T0) is back in production. | ||
* GDSS743 (atlasStripInput - D1T0) is back in production. | * GDSS743 (atlasStripInput - D1T0) is back in production. | ||
− | + | * GDSS757 (cmsDisk - D1T0) is back in production. | |
<!-- ***************************************************** -----> | <!-- ***************************************************** -----> | ||
Line 47: | Line 47: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | ||
|} | |} | ||
− | |||
* GDSS756 (cmsDisk - D1T0) not in production. | * GDSS756 (cmsDisk - D1T0) not in production. | ||
<!-- ***************End Ongoing Disk Server Issues**************** -----> | <!-- ***************End Ongoing Disk Server Issues**************** -----> | ||
Line 70: | Line 69: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting. | ||
|} | |} | ||
− | + | * The ATLAS quota on Echo was increased by 500TB yesterday (2nd Jan). It now stands at 4.6PBytes. | |
+ | * A round of Linux kernel patching of back-end database systems has been completed. | ||
<!-- *************End Notable Changes made this last week************** -----> | <!-- *************End Notable Changes made this last week************** -----> | ||
<!-- ****************************************************************** -----> | <!-- ****************************************************************** -----> | ||
Line 81: | Line 81: | ||
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report. | | style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting since the last report. | ||
|} | |} | ||
− | No downtime scheduled in the GOCDB between 2017-12- | + | No downtime scheduled in the GOCDB between 2017-12-20 and 2018-01-03 |
<!-- **********************End GOC DB Entries************************** -----> | <!-- **********************End GOC DB Entries************************** -----> | ||
<!-- ****************************************************************** -----> | <!-- ****************************************************************** -----> | ||
Line 110: | Line 110: | ||
'''Listing by category:''' | '''Listing by category:''' | ||
* Castor: | * Castor: | ||
− | ** Update systems | + | ** Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done) |
** Move to generic Castor headnodes. | ** Move to generic Castor headnodes. | ||
* Echo: | * Echo: | ||
Line 119: | Line 119: | ||
* Internal | * Internal | ||
** DNS servers will be rolled out within the Tier1 network. | ** DNS servers will be rolled out within the Tier1 network. | ||
+ | ** Infrastructure | ||
+ | * Testing of power distribution boards in the R89 machine room is being scheduled for some time in March. The effect of this on our services is being discussed. | ||
<!-- ***************End Advanced warning for other interventions*************** -----> | <!-- ***************End Advanced warning for other interventions*************** -----> | ||
<!-- ************************************************************************** -----> | <!-- ************************************************************************** -----> | ||
Line 250: | Line 252: | ||
| 22/12/17 || 100 || 100 || 100 || 100 || 100|| 100 || | | 22/12/17 || 100 || 100 || 100 || 100 || 100|| 100 || | ||
|- | |- | ||
− | | 23/12/17 || 100 || 100 || 100 || 100 || style="background-color: yellow;" |53 || 100 || | + | | 23/12/17 || 100 || 100 || 100 || 100 || style="background-color: yellow;" |53 || 100 || Problem with VOs monitoring. |
|- | |- | ||
| 24/12/17 || 100 || 100 || 100 || style="background-color: yellow;" |98 || 100 || 100 || | | 24/12/17 || 100 || 100 || 100 || style="background-color: yellow;" |98 || 100 || 100 || | ||
Line 292: | Line 294: | ||
| 22/12/17 || 100 || style="background-color: grey;" | 0|| style="background-color: yellow;" | 98 || Atlas HC Echo - No test run in time bin | | 22/12/17 || 100 || style="background-color: grey;" | 0|| style="background-color: yellow;" | 98 || Atlas HC Echo - No test run in time bin | ||
|- | |- | ||
− | | 23/12/17 || | + | | 23/12/17 || 100|| style="background-color: grey;" | 0 || 100 || Atlas HC Echo - No test run in time bin |
|- | |- | ||
| 24/12/17 || style="background-color: red;" | 0|| style="background-color: grey;" | 0 || 100 || Atlas HC Echo - No test run in time bin | | 24/12/17 || style="background-color: red;" | 0|| style="background-color: grey;" | 0 || 100 || Atlas HC Echo - No test run in time bin | ||
|- | |- | ||
− | | 25/12/17 || | + | | 25/12/17 || 100 || style="background-color: grey;" | 0 || 100 || Atlas HC Echo - No test run in time bin |
|- | |- | ||
| 26/12/17 || 100||style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin | | 26/12/17 || 100||style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin | ||
Line 306: | Line 308: | ||
| 29/12/17 || 100||style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin | | 29/12/17 || 100||style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin | ||
|- | |- | ||
− | | 30/12/17 || style="background-color: yellow;" | 93 ||style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin | + | | 30/12/17 || style="background-color: yellow;" | 93 || style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin |
|- | |- | ||
| 31/12/17 || 100||style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin | | 31/12/17 || 100||style="background-color: grey;" | 0|| 100 || Atlas HC Echo - No test run in time bin | ||
Line 324: | Line 326: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting. | ||
|} | |} | ||
− | * | + | * CMS quota on Echo will be increased from 250 to 500TB. CMS will then do more stress testing. |
+ | * MICE have probably taken their final data. Some problems with rate of transfer of data into Castor will no longer be relevant. |
Latest revision as of 14:41, 3 January 2018
RAL Tier1 Operations Report for 3rd January 2018
Review of Issues during the week 21st December 2017 to 3rd January 2018 |
- Network connectivity issues on Stack 9 in the UPS room overnight Thursday/Friday 21/22 December affected some internal systems (mainly monitoring). A member of staff attended overnight and a faulty transceiver was found the be the cause and replaced. External services were unaffected.
- Operations over the Christmas and New year holiday were generally stable although not completely quiet for the oncall team. There were some Castor disk server failures and staff did attend site over the holiday to replace failed disk drives. Our monitoring flagged problems with LHCb SAM tests - although these turned out to be a caused by a problem with a LHCb certificate.
- The number of Atlas batch jobs being run is lower than expected. The batch (Condor) scheduling will be looked at to try and understand and improve this.
Current operational status and issues |
- None
Resolved Castor Disk Server Issues |
- GDSS688 (cmsDisk - D1T0) is back in production.
- GDSS743 (atlasStripInput - D1T0) is back in production.
- GDSS757 (cmsDisk - D1T0) is back in production.
Ongoing Castor Disk Server Issues |
- GDSS756 (cmsDisk - D1T0) not in production.
Limits on concurrent batch system jobs. |
- CMS Multicore 550
Notable Changes made since the last meeting. |
- The ATLAS quota on Echo was increased by 500TB yesterday (2nd Jan). It now stands at 4.6PBytes.
- A round of Linux kernel patching of back-end database systems has been completed.
Entries in GOC DB starting since the last report. |
No downtime scheduled in the GOCDB between 2017-12-20 and 2018-01-03
Declared in the GOC DB |
- None
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Ongoing or Pending - but not yet formally announced:
Listing by category:
- Castor:
- Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
- Move to generic Castor headnodes.
- Echo:
- Update to next CEPH version ("Luminous").
- Networking
- Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
- Services
- Internal
- DNS servers will be rolled out within the Tier1 network.
- Infrastructure
- Testing of power distribution boards in the R89 machine room is being scheduled for some time in March. The effect of this on our services is being discussed.
Open GGUS Tickets (Snapshot during morning of meeting) |
Ticket-ID | Type | VO | Notified Site | Resp. Unit | Status | Priority | Creation | Last Update | ToI | Subject |
---|---|---|---|---|---|---|---|---|---|---|
132589 | TEAM | lhcb | RAL-LCG2 | NGI_UK | in progress | very urgent | 2017-12-21 06:45:00 | 2017-12-21 16:22:00 | Local Batch System | Killed pilots at RAL |
132540 | TEAM | lhcb | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | in progress | top priority | 2017-12-18 09:32:00 | 2017-12-23 10:13:00 | Other | Upload problems at RAL |
131815 | USER | t2k.org | RAL-LCG2 | NGI_UK | in progress | less urgent | 2017-11-13 14:42:00 | 2017-12-01 19:30:00 | Storage Systems | Extremely long download times for T2K files on tape at RAL |
130207 | USER | mice | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | on hold | urgent | 2017-08-24 09:46:00 | 2017-12-18 17:22:00 | Network problem | Timeouts when copyiing MICE reco data to CASTOR |
127597 | USER | cms | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk share with:sexton@fnal.gov | on hold | urgent | 2017-04-07 10:34:00 | 2017-10-05 09:14:00 | File Transfer | Check networking and xrootd RAL-CERN performance |
124876 | USER | ops | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | on hold | less urgent | 2016-11-07 12:06:00 | 2017-11-13 16:55:00 | Operations | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk |
117683 | USER | none | RAL-LCG2 | NGI_UK assign to:lcg-support@gridpp.rl.ac.uk | on hold | less urgent | 2015-11-18 11:36:00 | 2017-11-06 16:59:00 | Information System | CASTOR at RAL not publishing GLUE 2 |
Availability Report |
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas Echo | Comment |
---|---|---|---|---|---|---|---|
20/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
21/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
22/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
23/12/17 | 100 | 100 | 100 | 100 | 53 | 100 | Problem with VOs monitoring. |
24/12/17 | 100 | 100 | 100 | 98 | 100 | 100 | |
25/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
26/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
27/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
28/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
29/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
30/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
31/12/17 | 100 | 100 | 100 | 100 | 100 | 100 | |
01/01/18 | 100 | 100 | 100 | 100 | 100 | 100 | |
02/01/18 | 100 | 100 | 100 | 100 | 100 | 100 |
Hammercloud Test Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC Echo = Atlas Echo (Template 841);CMS HC = CMS HammerCloud
Day | Atlas HC | Atlas HC Echo | CMS HC | Comment |
---|---|---|---|---|
20/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
21/12/17 | 98 | 0 | 100 | Atlas HC Echo - No test run in time bin |
22/12/17 | 100 | 0 | 98 | Atlas HC Echo - No test run in time bin |
23/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
24/12/17 | 0 | 0 | 100 | Atlas HC Echo - No test run in time bin |
25/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
26/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
27/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
28/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
29/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
30/12/17 | 93 | 0 | 100 | Atlas HC Echo - No test run in time bin |
31/12/17 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
01/01/18 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
02/01/18 | 100 | 0 | 100 | Atlas HC Echo - No test run in time bin |
Notes from Meeting. |
- CMS quota on Echo will be increased from 250 to 500TB. CMS will then do more stress testing.
- MICE have probably taken their final data. Some problems with rate of transfer of data into Castor will no longer be relevant.