Difference between revisions of "Tier1 Operations Report 2018-08-06"
From GridPP Wiki
(→) |
(→) |
||
Line 236: | Line 236: | ||
! Scope | ! Scope | ||
|- | |- | ||
− | | style="background-color: green;" | | + | | style="background-color: green;" | 136537 |
− | | | + | | ops |
+ | | verified | ||
+ | | less urgent | ||
+ | | 03/08/2018 | ||
+ | | 06/08/2018 | ||
+ | | Operations | ||
+ | | [Rod Dashboard] Issue detected : org.bdii.GLUE2-Validate@site-bdii.gridpp.rl.ac.uk | ||
+ | | EGI | ||
+ | |- | ||
+ | | 136460 | ||
+ | | cms | ||
| solved | | solved | ||
| urgent | | urgent | ||
− | | | + | | 30/07/2018 |
− | | | + | | 01/08/2018 |
− | | | + | | CMS_Data Transfers |
− | | | + | | Transfers failing to RAL_Buffer |
| WLCG | | WLCG | ||
|- | |- | ||
− | | | + | | 136408 |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
| cms | | cms | ||
| solved | | solved | ||
| urgent | | urgent | ||
− | |||
| 27/07/2018 | | 27/07/2018 | ||
− | | | + | | 01/08/2018 |
− | | | + | | CMS_Data Transfers |
+ | | missing files at RAL | ||
| WLCG | | WLCG | ||
|- | |- | ||
− | | | + | | 136229 |
| cms | | cms | ||
− | | | + | | closed |
− | | urgent | + | | very urgent |
− | | | + | | 19/07/2018 |
− | | | + | | 02/08/2018 |
− | | | + | | Data Management - generic |
− | | | + | | RAL FTS Service not reachable via IPv6 |
− | | | + | | EGI |
|- | |- | ||
| 136138 | | 136138 | ||
| t2k.org | | t2k.org | ||
− | | | + | | verified |
| urgent | | urgent | ||
| 16/07/2018 | | 16/07/2018 | ||
− | | | + | | 02/08/2018 |
| File Access | | File Access | ||
| Extremely long download times for T2K files on tape at RALL - Part 2 | | Extremely long download times for T2K files on tape at RALL - Part 2 | ||
| EGI | | EGI | ||
|- | |- | ||
− | | | + | | 136110 |
− | | | + | | atlas |
− | | | + | | closed |
− | | | + | | urgent |
| 13/07/2018 | | 13/07/2018 | ||
− | | | + | | 31/07/2018 |
− | | | + | | File Transfer |
− | | | + | | RAL-LCG2: Transfer errors as source with "SRM_FILE_UNAVAILABLE" |
− | | | + | | WLCG |
|- | |- | ||
− | | | + | | 136097 |
− | | | + | | other |
| closed | | closed | ||
| urgent | | urgent | ||
− | | | + | | 13/07/2018 |
− | | | + | | 03/08/2018 |
− | | | + | | Operations |
− | | | + | | Please restart frontier-squid on RAL cvmfs stratum 1 |
− | | | + | | EGI |
|} | |} | ||
<!-- **********************End Availability Report************************** -----> | <!-- **********************End Availability Report************************** -----> |
Revision as of 07:25, 7 August 2018
RAL Tier1 Operations Report for 6th August 2018
Review of Issues during the week 30th July to the 6th August 2018. |
- Every circuit breaker located in Tier-1's R89 building was tested between Tuesday 24th - Thursday 26th July. This testing period was completed successfully. The majority of racks were powered by dual PDUs which were on different circuit breakers and so theoretically, nothing should have gone down. In reality 23 PDUs failed over the 3 day period. This was within the expected number of failures although it was at the highest end of our estimates. We had sufficient replacement PDUs.
- Impact to Tier-1:
- 2 Production Castor disk servers went down with temporary lost of access to some file access.
- Castor transfer manager went down for ATLAS, lost of ability to schedule new transfers for ~20 minutes.
- A few SL5 machines also went down. No impact on service but a useful reminder to get them decommissioned properly.
- It should be worth noting that two Echo storage nodes went down with no impact on service. This happened because their power cables weren’t correctly fitted.
- GOCDB service was down for 45 minutes. It wasn’t directly a result of the power testing it was failed network switch (GOCDB is not on Tier-1 network, so this failure did not impact the Tier-1). The read only version (in Germany) stayed up. EGI broadcast was sent out a few minutes after the service went down.
Current operational status and issues |
- None.
Resolved Castor Disk Server Issues |
- None
Ongoing Castor Disk Server Issues |
- None.
Limits on concurrent batch system jobs. |
- CMS Multicore 550
Notable Changes made since the last meeting. |
- None.
Entries in GOC DB starting since the last report. |
Declared in the GOC DB |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Listing by category:
- Castor:
- Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
- Move to generic Castor headnodes.
- Internal
- DNS servers will be rolled out within the Tier1 network.
Open
GGUS Tickets (Snapshot taken during morning of the meeting). |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
136563 | cms | in progress | urgent | 06/08/2018 | 06/08/2018 | CMS_Data Transfers | Possibly corrupted files at RAL | WLCG |
136366 | mice | in progress | less urgent | 25/07/2018 | 26/07/2018 | Local Batch System | Remove MICE Queue from RAL T1 Batch | EGI |
136358 | cms | on hold | urgent | 25/07/2018 | 03/08/2018 | CMS_Facilities | T1_UK_RAL WN-xrootd-access failure | WLCG |
136199 | lhcb | in progress | very urgent | 18/07/2018 | 06/08/2018 | File Transfer | Lots of submitted transfers on RAL FTS | WLCG |
136028 | cms | in progress | top priority | 10/07/2018 | 06/08/2018 | CMS_AAA WAN Access | Issues reading files at T1_UK_RAL_Disk | WLCG |
134685 | dteam | waiting for reply | less urgent | 23/04/2018 | 06/08/2018 | Middleware | please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 | EGI |
124876 | ops | in progress | less urgent | 07/11/2016 | 23/07/2018 | Operations | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk | EGI |
GGUS Tickets Closed Last week |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
136537 | ops | verified | less urgent | 03/08/2018 | 06/08/2018 | Operations | [Rod Dashboard] Issue detected : org.bdii.GLUE2-Validate@site-bdii.gridpp.rl.ac.uk | EGI |
136460 | cms | solved | urgent | 30/07/2018 | 01/08/2018 | CMS_Data Transfers | Transfers failing to RAL_Buffer | WLCG |
136408 | cms | solved | urgent | 27/07/2018 | 01/08/2018 | CMS_Data Transfers | missing files at RAL | WLCG |
136229 | cms | closed | very urgent | 19/07/2018 | 02/08/2018 | Data Management - generic | RAL FTS Service not reachable via IPv6 | EGI |
136138 | t2k.org | verified | urgent | 16/07/2018 | 02/08/2018 | File Access | Extremely long download times for T2K files on tape at RALL - Part 2 | EGI |
136110 | atlas | closed | urgent | 13/07/2018 | 31/07/2018 | File Transfer | RAL-LCG2: Transfer errors as source with "SRM_FILE_UNAVAILABLE" | WLCG |
136097 | other | closed | urgent | 13/07/2018 | 03/08/2018 | Operations | Please restart frontier-squid on RAL cvmfs stratum 1 | EGI |
Availability Report |
Target Availability for each site is 97.0% | Red <90% | Orange <97% |
Day | Atlas | Atlas-Echo | CMS | LHCB | Alice | OPS | Comments |
---|---|---|---|---|---|---|---|
2018-07-23 | 100 | 100 | 100 | 100 | 100 | 100 | |
2018-07-24 | 100 | 100 | 97 | 100 | 100 | 100 | |
2018-07-25 | 98 | 100 | 93 | 100 | 100 | 100 | |
2018-07-26 | 100 | 100 | 100 | 100 | 100 | 100 | |
2018-07-27 | 100 | 100 | 99 | 100 | 100 | 100 | |
2018-07-28 | 100 | 100 | 96 | 100 | 84 | 100 | |
2018-07-29 | 100 | 100 | 98 | 100 | 100 | 100 |
Hammercloud Test Report |
Target Availability for each site is 97.0% | Red <90% | Orange <97% |
Day | Atlas HC | CMS HC | Comment |
---|---|---|---|
2018-07-24 | 95 | 99 | |
2018-07-25 | 98 | 100 | |
2018-07-26 | 98 | 99 | |
2018-07-27 | 98 | 99 | |
2018-07-28 | 98 | 98 | |
2018-07-29 | 100 | 98 | |
2018-07-30 | - | - |
Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud
Notes from Meeting. |