Difference between revisions of "Tier1 Operations Report 2018-06-11"
From GridPP Wiki
(→) |
(→) |
||
(14 intermediate revisions by 2 users not shown) | |||
Line 22: | Line 22: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | ||
|} | |} | ||
− | * | + | * '''Continuing problems with small VOs and Castor (BDii)''' - Please be advised that in the light of the impact on a number of smaller VO's the removal of this service has caused, that Castor information (at least in the short term), is now being published in the BDii. |
<!-- ***********End Current operational status and issues*********** -----> | <!-- ***********End Current operational status and issues*********** -----> | ||
<!-- *************************************************************** -----> | <!-- *************************************************************** -----> | ||
Line 33: | Line 33: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues | ||
|} | |} | ||
− | * | + | * gdss738 - LHCb - d1t0. Currently back in production RO. |
− | + | ||
− | + | ||
<!-- ***************************************************** -----> | <!-- ***************************************************** -----> | ||
Line 45: | Line 43: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | ||
|} | |} | ||
− | * | + | * gdss687 - LHCb - d1t0. Currently still in intervention. |
<!-- ***************End Ongoing Disk Server Issues**************** -----> | <!-- ***************End Ongoing Disk Server Issues**************** -----> | ||
<!-- ************************************************************* -----> | <!-- ************************************************************* -----> | ||
Line 124: | Line 122: | ||
|- | |- | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open | ||
− | GGUS Tickets (Snapshot taken during morning of the | + | GGUS Tickets (Snapshot taken during morning of the meeting). |
|} | |} | ||
{| border=1 align=center | {| border=1 align=center | ||
Line 300: | Line 298: | ||
! Alice | ! Alice | ||
! OPS | ! OPS | ||
− | ! Comments | + | ! Comments |
|- | |- | ||
− | | 2018- | + | | 2018-06-04 |
− | + | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
+ | | 97 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
Line 311: | Line 309: | ||
| | | | ||
|- | |- | ||
− | | 2018- | + | | 2018-06-05 |
| 100 | | 100 | ||
| 100 | | 100 | ||
+ | | 99 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | + | | style="background-color: red;" | 46.80851 | |
− | | style="background-color: red;" | | + | | Would hazzard a guess that BDii has just been put back! |
− | | | + | |
|- | |- | ||
− | | 2018- | + | | 2018-06-06 |
− | | | + | | 100 |
+ | | 100 | ||
+ | | style="background-color: orange;" | 91 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | |||
− | |||
| | | | ||
|- | |- | ||
− | | 2018- | + | | 2018-06-07 |
| 100 | | 100 | ||
| 100 | | 100 | ||
+ | | 99 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | |||
| | | | ||
|- | |- | ||
− | | 2018-06- | + | | 2018-06-08 |
+ | | 100 | ||
+ | | 100 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | |||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | |||
| | | | ||
|- | |- | ||
− | | 2018-06- | + | | 2018-06-09 |
− | | | + | | 100 |
+ | | 100 | ||
+ | | 99 | ||
| 100 | | 100 | ||
− | |||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | |||
| | | | ||
|- | |- | ||
− | | 2018-06- | + | | 2018-06-10 |
| 100 | | 100 | ||
| 100 | | 100 | ||
− | | | + | | 97 |
+ | | 100 | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | |||
| | | | ||
|} | |} | ||
Line 382: | Line 380: | ||
! Day !! Atlas HC !! CMS HC !! Comment | ! Day !! Atlas HC !! CMS HC !! Comment | ||
|- | |- | ||
− | | 2018/ | + | | 2018/06/04 || style="background-color: orange;" | 95 || style="background-color: red;" |39|| |
|- | |- | ||
− | | 2018/ | + | | 2018/06/05 || style="background-color: orange;" | 94 || style="background-color: red;" |19 || |
|- | |- | ||
− | | 2018/ | + | | 2018/06/06 || style="background-color: red;" | 89 || style="background-color: red;" |84 || |
|- | |- | ||
− | | 2018/06/ | + | | 2018/06/07 || style="background-color: red;" | 89 || 100 || |
|- | |- | ||
− | | 2018/06/ | + | | 2018/06/08 || style="background-color: orange;" | 95 || style="background-color: red;" |89 || |
|- | |- | ||
− | | 2018/06/ | + | | 2018/06/09 || style="background-color: red;" | 87 || style="background-color: red;" |66 || |
|- | |- | ||
− | | 2018/06/ | + | | 2018/06/10 || style="background-color: red;" | 89 || style="background-color: red;" |66 || |
+ | |- | ||
+ | | 2018/06/11 || - || - || | ||
|- | |- | ||
|} | |} | ||
Line 408: | Line 408: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting. | ||
|} | |} | ||
− | * | + | * Notes from meeting of 13th June: |
+ | |||
+ | * SKA: | ||
+ | ** It was noted that the SKA batch jobs are limited to 24 cores by the Dirac system (at Imperial) that they are using. | ||
+ | * Dune/ProtoDune: | ||
+ | ** There is now a regular monday meeting for Dune/ProtoDune. Raja, as the Dune-Tier1 liaison person will attend regularly. Darren will attend in his place if Raja cannot be there. | ||
+ | ** Access to the Tier1 batch farm is being set-up. The Tier1's aim is to get this set-up for initial testing by next Monday's meeting. (18th June). | ||
+ | ** Alastair has set-up Dune access to Echo (line for access via WebDAV provided to Raja). | ||
+ | * Euclid: | ||
+ | ** We are in contact with Euclid to get the public key for access to the Euclid CVMFS repository. | ||
+ | |||
+ | Documentation to Get VOs started accessing Tier1 Services: | ||
+ | A start has been made on this. See: https://www.gridpp.ac.uk/wiki/RAL_Tier1_Echo |
Latest revision as of 14:52, 13 June 2018
RAL Tier1 Operations Report for 11th June 2018
Review of Issues during the week 28th May to the 4th June 2018. |
- No incidents(major or minor), have been flagged during this reporting period.
Current operational status and issues |
- Continuing problems with small VOs and Castor (BDii) - Please be advised that in the light of the impact on a number of smaller VO's the removal of this service has caused, that Castor information (at least in the short term), is now being published in the BDii.
Resolved Castor Disk Server Issues |
- gdss738 - LHCb - d1t0. Currently back in production RO.
Ongoing Castor Disk Server Issues |
- gdss687 - LHCb - d1t0. Currently still in intervention.
Limits on concurrent batch system jobs. |
- CMS Multicore 550
Notable Changes made since the last meeting. |
- None.
Entries in GOC DB starting since the last report. |
Declared in the GOC DB |
- None
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Listing by category:
- Castor:
- Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
- Move to generic Castor headnodes.
- Networking
- Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
- Internal
- DNS servers will be rolled out within the Tier1 network.
- Infrastructure
- Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.
Open
GGUS Tickets (Snapshot taken during morning of the meeting). |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
135455 | cms | in progress | less urgent | 31/05/2018 | 04/06/2018 | File Transfer | Checksum verification at RAL | EGI |
135367 | snoplus.snolab.ca | in progress | less urgent | 28/05/2018 | 08/06/2018 | Other | Lost access to srm-snoplus.gridpp.rl.ac.uk | EGI |
135308 | mice | in progress | top priority | 24/05/2018 | 01/06/2018 | Information System | Can't send data to RAL Castor | EGI |
135293 | ops | on hold | less urgent | 23/05/2018 | 04/06/2018 | Operations | [Rod Dashboard] Issues detected at RAL-LCG2 | EGI |
134685 | dteam | in progress | less urgent | 23/04/2018 | 02/05/2018 | Middleware | please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 | EGI |
124876 | ops | on hold | less urgent | 07/11/2016 | 13/11/2017 | Operations | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk | EGI |
GGUS Tickets Closed Last week |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
135516 | cms | solved | urgent | 06/06/2018 | 08/06/2018 | CMS_Facilities | T1_UK_RAL WN-analysis and HammerCloud failures | WLCG |
135133 | cms | solved | urgent | 15/05/2018 | 05/06/2018 | CMS_Data Transfers | Likely corrupted File at T1_UK_RAL | WLCG |
134703 | cms | solved | urgent | 23/04/2018 | 07/06/2018 | CMS_Data Transfers | Transfer failing from RAL_Disk | WLCG |
133992 | atlas | verified | less urgent | 12/03/2018 | 04/06/2018 | File Transfer | RAL-LCG2-ECHO: No such file or directory | EGI |
127597 | cms | solved | urgent | 07/04/2017 | 07/06/2018 | File Transfer | Check networking and xrootd RAL-CERN performance | EGI |
Availability Report |
Target Availability for each site is 97.0% | Red <90% | Orange <97% |
Day | Atlas | Atlas-Echo | CMS | LHCB | Alice | OPS | Comments |
---|---|---|---|---|---|---|---|
2018-06-04 | 100 | 100 | 97 | 100 | 100 | 0 | |
2018-06-05 | 100 | 100 | 99 | 100 | 100 | 46.80851 | Would hazzard a guess that BDii has just been put back! |
2018-06-06 | 100 | 100 | 91 | 100 | 100 | 100 | |
2018-06-07 | 100 | 100 | 99 | 100 | 100 | 100 | |
2018-06-08 | 100 | 100 | 100 | 100 | 100 | 100 | |
2018-06-09 | 100 | 100 | 99 | 100 | 100 | 100 | |
2018-06-10 | 100 | 100 | 97 | 100 | 100 | 100 |
Hammercloud Test Report |
Target Availability for each site is 97.0% | Red <90% | Orange <97% |
Day | Atlas HC | CMS HC | Comment |
---|---|---|---|
2018/06/04 | 95 | 39 | |
2018/06/05 | 94 | 19 | |
2018/06/06 | 89 | 84 | |
2018/06/07 | 89 | 100 | |
2018/06/08 | 95 | 89 | |
2018/06/09 | 87 | 66 | |
2018/06/10 | 89 | 66 | |
2018/06/11 | - | - |
Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud
Notes from Meeting. |
- Notes from meeting of 13th June:
- SKA:
- It was noted that the SKA batch jobs are limited to 24 cores by the Dirac system (at Imperial) that they are using.
- Dune/ProtoDune:
- There is now a regular monday meeting for Dune/ProtoDune. Raja, as the Dune-Tier1 liaison person will attend regularly. Darren will attend in his place if Raja cannot be there.
- Access to the Tier1 batch farm is being set-up. The Tier1's aim is to get this set-up for initial testing by next Monday's meeting. (18th June).
- Alastair has set-up Dune access to Echo (line for access via WebDAV provided to Raja).
- Euclid:
- We are in contact with Euclid to get the public key for access to the Euclid CVMFS repository.
Documentation to Get VOs started accessing Tier1 Services: A start has been made on this. See: https://www.gridpp.ac.uk/wiki/RAL_Tier1_Echo