Difference between revisions of "Tier1 Operations Report 2018-06-04"
From GridPP Wiki
(→) |
(→) |
||
(19 intermediate revisions by one user not shown) | |||
Line 22: | Line 22: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | ||
|} | |} | ||
− | * | + | * For the avoidance of doubt regards confusion over the status of both the BDii and Castor services. Tier-1 have NOT stopped or decommissioned these services. What has changed is that the Castor information stopped getting published in the BDii. This was due to a failure of the machine running the Castor Information Provide (CIP). The CIP was a legacy part of Castor (It didn't support GLUE2 schema), and was due to be decommissioned, the hardware failure expedited this issue. Although we admit this is not ideal, we decided against bringing back a service simply to then inform people we would shortly remove it. |
<!-- ***********End Current operational status and issues*********** -----> | <!-- ***********End Current operational status and issues*********** -----> | ||
<!-- *************************************************************** -----> | <!-- *************************************************************** -----> | ||
Line 33: | Line 33: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Castor Disk Server Issues | ||
|} | |} | ||
− | * | + | * gdss801 - cmsDisk - d1t0. Currently back in production RO. |
+ | * gdss817 - LHCb - d1t0 - Currently back in production RO. | ||
+ | * gdss733 - LHCb - d1t0 - Currently back in production RO. | ||
<!-- ***************************************************** -----> | <!-- ***************************************************** -----> | ||
Line 43: | Line 45: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Castor Disk Server Issues | ||
|} | |} | ||
− | * None | + | * None. |
<!-- ***************End Ongoing Disk Server Issues**************** -----> | <!-- ***************End Ongoing Disk Server Issues**************** -----> | ||
<!-- ************************************************************* -----> | <!-- ************************************************************* -----> | ||
Line 122: | Line 124: | ||
|- | |- | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Open | ||
− | GGUS Tickets (Snapshot during morning of the report). | + | GGUS Tickets (Snapshot taken during morning of the report). |
|} | |} | ||
{| border=1 align=center | {| border=1 align=center | ||
Line 136: | Line 138: | ||
! Scope | ! Scope | ||
|- | |- | ||
− | | style="background-color: green;" | 135455 | + | | style="background-color: green;" |135516 |
+ | | cms | ||
+ | | in progress | ||
+ | | urgent | ||
+ | | 06/06/2018 | ||
+ | | 06/06/2018 | ||
+ | | CMS_Facilities | ||
+ | | T1_UK_RAL WN-analysis and HammerCloud failures | ||
+ | | WLCG | ||
+ | |- | ||
+ | | style="background-color: green;" |135455 | ||
| cms | | cms | ||
| in progress | | in progress | ||
| less urgent | | less urgent | ||
| 31/05/2018 | | 31/05/2018 | ||
− | | | + | | 04/06/2018 |
| File Transfer | | File Transfer | ||
| Checksum verification at RAL | | Checksum verification at RAL | ||
| EGI | | EGI | ||
|- | |- | ||
− | | style="background-color: green;" | 135308 | + | | style="background-color: green;" |135367 |
+ | | snoplus.snolab.ca | ||
+ | | in progress | ||
+ | | less urgent | ||
+ | | 28/05/2018 | ||
+ | | 05/06/2018 | ||
+ | | Other | ||
+ | | Lost access to srm-snoplus.gridpp.rl.ac.uk | ||
+ | | EGI | ||
+ | |- | ||
+ | | style="background-color: green;" |135308 | ||
| mice | | mice | ||
| in progress | | in progress | ||
Line 156: | Line 178: | ||
| EGI | | EGI | ||
|- | |- | ||
− | | style="background-color: green;" | 135293 | + | | style="background-color: green;" |135293 |
| ops | | ops | ||
| on hold | | on hold | ||
| less urgent | | less urgent | ||
| 23/05/2018 | | 23/05/2018 | ||
− | | | + | | 04/06/2018 |
| Operations | | Operations | ||
| [Rod Dashboard] Issues detected at RAL-LCG2 | | [Rod Dashboard] Issues detected at RAL-LCG2 | ||
| EGI | | EGI | ||
|- | |- | ||
− | | style="background-color: | + | | style="background-color: yellow;" |134703 |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
| cms | | cms | ||
| in progress | | in progress | ||
Line 186: | Line 198: | ||
| WLCG | | WLCG | ||
|- | |- | ||
− | | style="background-color: green;" | 134685 | + | | style="background-color: green;" |134685 |
| dteam | | dteam | ||
| in progress | | in progress | ||
Line 196: | Line 208: | ||
| EGI | | EGI | ||
|- | |- | ||
− | + | | style="background-color: red;" |127597 | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | | style="background-color: red;" | 127597 | + | |
| cms | | cms | ||
| on hold | | on hold | ||
Line 216: | Line 218: | ||
| EGI | | EGI | ||
|- | |- | ||
− | | style="background-color: red;" | 124876 | + | | style="background-color: red;" |124876 |
| ops | | ops | ||
| on hold | | on hold | ||
Line 259: | Line 261: | ||
| [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability | | [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability | ||
| EGI | | EGI | ||
+ | |- | ||
+ | | 135133 | ||
+ | | cms | ||
+ | | solved | ||
+ | | urgent | ||
+ | | 15/05/2018 | ||
+ | | 05/06/2018 | ||
+ | | CMS_Data Transfers | ||
+ | | Likely corrupted File at T1_UK_RAL | ||
+ | | WLCG | ||
|- | |- | ||
| 134468 | | 134468 | ||
Line 269: | Line 281: | ||
| Xrootd redirector not seeing some files in ECHO | | Xrootd redirector not seeing some files in ECHO | ||
| WLCG | | WLCG | ||
+ | |- | ||
+ | | 133992 | ||
+ | | atlas | ||
+ | | verified | ||
+ | | less urgent | ||
+ | | 12/03/2018 | ||
+ | | 04/06/2018 | ||
+ | | File Transfer | ||
+ | | RAL-LCG2-ECHO: No such file or directory | ||
+ | | EGI | ||
|- | |- | ||
| 117683 | | 117683 | ||
Line 279: | Line 301: | ||
| CASTOR at RAL not publishing GLUE 2 | | CASTOR at RAL not publishing GLUE 2 | ||
| EGI | | EGI | ||
− | |}<!-- **********************End Availability Report************************** -----> | + | |} |
+ | <!-- **********************End Availability Report************************** -----> | ||
<!-- *********************************************************************** -----> | <!-- *********************************************************************** -----> | ||
<!-- **********************End GGUS Tickets************************** -----> | <!-- **********************End GGUS Tickets************************** -----> | ||
Line 357: | Line 380: | ||
| 98 | | 98 | ||
| 100 | | 100 | ||
− | + | | 97 | |
| 100 | | 100 | ||
| 100 | | 100 | ||
Line 385: | Line 408: | ||
| style="background-color: orange;" | Orange <97% | | style="background-color: orange;" | Orange <97% | ||
|} | |} | ||
− | |||
{| border=1 align=center | {| border=1 align=center | ||
|- bgcolor="#7c8aaf" | |- bgcolor="#7c8aaf" | ||
! Day !! Atlas HC !! CMS HC !! Comment | ! Day !! Atlas HC !! CMS HC !! Comment | ||
|- | |- | ||
− | | 2018/05/ | + | | 2018/05/29 || 97|| 99|| |
|- | |- | ||
− | | 2018/05/ | + | | 2018/05/30 || style="background-color: Red;" |95 || 100 || |
|- | |- | ||
− | | 2018/05/ | + | | 2018/05/31 || 97 || style="background-color: Red;" | 71 || |
|- | |- | ||
− | | 2018/ | + | | 2018/06/01 || style="background-color: orange;" | 96 || 98 || |
|- | |- | ||
− | | 2018/ | + | | 2018/06/02 || 97 || 98 || |
|- | |- | ||
− | | 2018/ | + | | 2018/06/03 || 98 || 98 || |
|- | |- | ||
− | | 2018/ | + | | 2018/06/04 || style="background-color: orange;" | 95 || style="background-color: Red;" |0 || |
|- | |- | ||
|} | |} | ||
+ | Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud | ||
<!-- **********************End Hammercloud Test Report************************** -----> | <!-- **********************End Hammercloud Test Report************************** -----> | ||
<!-- *********************************************************************** -----> | <!-- *********************************************************************** -----> |
Latest revision as of 10:35, 6 June 2018
RAL Tier1 Operations Report for 4th June 2018
Review of Issues during the week 28th May to the 4th June 2018. |
- No incidents(major or minor), have been flagged during this reporting period.
Current operational status and issues |
- For the avoidance of doubt regards confusion over the status of both the BDii and Castor services. Tier-1 have NOT stopped or decommissioned these services. What has changed is that the Castor information stopped getting published in the BDii. This was due to a failure of the machine running the Castor Information Provide (CIP). The CIP was a legacy part of Castor (It didn't support GLUE2 schema), and was due to be decommissioned, the hardware failure expedited this issue. Although we admit this is not ideal, we decided against bringing back a service simply to then inform people we would shortly remove it.
Resolved Castor Disk Server Issues |
- gdss801 - cmsDisk - d1t0. Currently back in production RO.
- gdss817 - LHCb - d1t0 - Currently back in production RO.
- gdss733 - LHCb - d1t0 - Currently back in production RO.
Ongoing Castor Disk Server Issues |
- None.
Limits on concurrent batch system jobs. |
- CMS Multicore 550
Notable Changes made since the last meeting. |
- None.
Entries in GOC DB starting since the last report. |
Declared in the GOC DB |
- None
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Listing by category:
- Castor:
- Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
- Move to generic Castor headnodes.
- Networking
- Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
- Internal
- DNS servers will be rolled out within the Tier1 network.
- Infrastructure
- Testing of power distribution boards in the R89 machine room is being scheduled for some time late July / Early August. The effect of this on our services is being discussed.
Open
GGUS Tickets (Snapshot taken during morning of the report). |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
135516 | cms | in progress | urgent | 06/06/2018 | 06/06/2018 | CMS_Facilities | T1_UK_RAL WN-analysis and HammerCloud failures | WLCG |
135455 | cms | in progress | less urgent | 31/05/2018 | 04/06/2018 | File Transfer | Checksum verification at RAL | EGI |
135367 | snoplus.snolab.ca | in progress | less urgent | 28/05/2018 | 05/06/2018 | Other | Lost access to srm-snoplus.gridpp.rl.ac.uk | EGI |
135308 | mice | in progress | top priority | 24/05/2018 | 01/06/2018 | Information System | Can't send data to RAL Castor | EGI |
135293 | ops | on hold | less urgent | 23/05/2018 | 04/06/2018 | Operations | [Rod Dashboard] Issues detected at RAL-LCG2 | EGI |
134703 | cms | in progress | urgent | 23/04/2018 | 25/05/2018 | CMS_Data Transfers | Transfer failing from RAL_Disk | WLCG |
134685 | dteam | in progress | less urgent | 23/04/2018 | 02/05/2018 | Middleware | please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 | EGI |
127597 | cms | on hold | urgent | 07/04/2017 | 30/04/2018 | File Transfer | Check networking and xrootd RAL-CERN performance | EGI |
124876 | ops | on hold | less urgent | 07/11/2016 | 13/11/2017 | Operations | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk | EGI |
GGUS Tickets Closed Last week |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
135342 | ops | unsolved | less urgent | 27/05/2018 | 31/05/2018 | Operations | [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability | EGI |
135133 | cms | solved | urgent | 15/05/2018 | 05/06/2018 | CMS_Data Transfers | Likely corrupted File at T1_UK_RAL | WLCG |
134468 | cms | solved | top priority | 09/04/2018 | 01/06/2018 | CMS_AAA WAN Access | Xrootd redirector not seeing some files in ECHO | WLCG |
133992 | atlas | verified | less urgent | 12/03/2018 | 04/06/2018 | File Transfer | RAL-LCG2-ECHO: No such file or directory | EGI |
117683 | none | unsolved | less urgent | 18/11/2015 | 31/05/2018 | Information System | CASTOR at RAL not publishing GLUE 2 | EGI |
Availability Report |
Target Availability for each site is 97.0% | Red <90% | Orange <97% |
Day | Atlas | Atlas-Echo | CMS | LHCB | Alice | OPS | Comments |
---|---|---|---|---|---|---|---|
2018-05-28 | 100 | 100 | 100 | 100 | 100 | 0 | |
2018-05-29 | 100 | 100 | 100 | 100 | 100 | 0 | |
2018-05-30 | 98 | 100 | 100 | 100 | 85 | 0 | |
2018-05-31 | 100 | 100 | 100 | 100 | 100 | 0 | |
2018-06-01 | 100 | 100 | 99 | 100 | 100 | 0 | |
2018-06-02 | 98 | 100 | 97 | 100 | 100 | 0 | |
2018-06-03 | 100 | 100 | 99 | 100 | 100 | 0 |
Hammercloud Test Report |
Target Availability for each site is 97.0% | Red <90% | Orange <97% |
Day | Atlas HC | CMS HC | Comment |
---|---|---|---|
2018/05/29 | 97 | 99 | |
2018/05/30 | 95 | 100 | |
2018/05/31 | 97 | 71 | |
2018/06/01 | 96 | 98 | |
2018/06/02 | 97 | 98 | |
2018/06/03 | 98 | 98 | |
2018/06/04 | 95 | 0 |
Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud
Notes from Meeting. |
- None yet