Difference between revisions of "Tier1 Operations Report 2018-10-15"
From GridPP Wiki
(→) |
(→) |
||
(3 intermediate revisions by one user not shown) | |||
Line 13: | Line 13: | ||
* No major issues of note to report for this week. | * No major issues of note to report for this week. | ||
* Occasional repeat of a CMS error – “500-globus_xio: System error in bind: Address already in use.” - This is not a major issue but does impact the SAM statistics. | * Occasional repeat of a CMS error – “500-globus_xio: System error in bind: Address already in use.” - This is not a major issue but does impact the SAM statistics. | ||
− | |||
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
<!-- *********************************************************** -----> | <!-- *********************************************************** -----> | ||
Line 24: | Line 23: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | ||
|} | |} | ||
− | * | + | * As an advisory Tier-1 is continuing with the current round of security patching (CVE-2018-14634, CVE-2018-8897 – SL7 and SL6 respectively). This involves all Tier-1 services but the risk of impact on availability is considered to be minimal and as such no declaration of service “At Risk” or “Outage” will be called. |
<!-- ***********End Current operational status and issues*********** -----> | <!-- ***********End Current operational status and issues*********** -----> | ||
<!-- *************************************************************** -----> | <!-- *************************************************************** -----> | ||
Line 179: | Line 178: | ||
* Internal | * Internal | ||
** DNS servers will be rolled out within the Tier1 network. | ** DNS servers will be rolled out within the Tier1 network. | ||
− | |||
<!-- ***************End Advanced warning for other interventions*************** -----> | <!-- ***************End Advanced warning for other interventions*************** -----> | ||
<!-- ************************************************************************** -----> | <!-- ************************************************************************** -----> | ||
Line 462: | Line 460: | ||
| 100 | | 100 | ||
| 100 | | 100 | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
| 100 | | 100 | ||
| 100 | | 100 |
Latest revision as of 09:55, 16 October 2018
RAL Tier1 Operations Report for 15th October 2018
Review of Issues during the week 8th October 2018 to the 15th October 2018. |
- No major issues of note to report for this week.
- Occasional repeat of a CMS error – “500-globus_xio: System error in bind: Address already in use.” - This is not a major issue but does impact the SAM statistics.
Current operational status and issues |
- As an advisory Tier-1 is continuing with the current round of security patching (CVE-2018-14634, CVE-2018-8897 – SL7 and SL6 respectively). This involves all Tier-1 services but the risk of impact on availability is considered to be minimal and as such no declaration of service “At Risk” or “Outage” will be called.
Resolved Castor Disk Server Issues |
Machine | VO | DiskPool | dxtx | Comments |
---|---|---|---|---|
gdss753 | Atlas | atlasStripInput | d1t0 | Decommissioned. |
Ongoing Castor Disk Server Issues |
Machine | VO | DiskPool | dxtx | Comments |
---|---|---|---|---|
gdss747 | Atlas | atlasStripInput | d1t0 | Currently in intervention. |
Limits on concurrent batch system jobs. |
- None currently enforced.
Notable Changes made since the last meeting. |
- None.
Entries in GOC DB starting since the last report. |
Service | ID | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|---|
- | - | - | - | - | - | - | - |
Declared in the GOC DB |
Service | ID | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|---|
- | - | - | - | - | - | - | - |
- No ongoing downtime
- No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Listing by category:
- Castor:
- Disk only storage will end in Castor. A new single tape-only Castor instance (called WLCGTape) is being tested. This is using generic Catsor headnodes on SL7 configured by Quattor/Aquilon with a slightly newer Castor version.
- Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
- Internal
- DNS servers will be rolled out within the Tier1 network.
Open
GGUS Tickets (Snapshot taken during morning of the meeting). |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
137650 | cms | waiting for reply | urgent | 09/10/2018 | 15/10/2018 | CMS_AAA WAN Access | Low HC xrootd success rates at T1_UK_RAL | WLCG |
137195 | ops | in progress | less urgent | 14/09/2018 | 15/10/2018 | Operations | [Rod Dashboard] Issues detected at RAL-LCG2 | EGI |
137153 | t2k.org | in progress | urgent | 12/09/2018 | 10/10/2018 | Data Management - generic | LFC entry has file size 0, preventsw registering of additional replicas | EGI |
136701 | lhcb | waiting for reply | very urgent | 14/08/2018 | 10/10/2018 | File Transfer | background of transfer errors | WLCG |
136199 | lhcb | waiting for reply | very urgent | 18/07/2018 | 11/10/2018 | File Transfer | Lots of submitted transfers on RAL FTS | WLCG |
124876 | ops | in progress | less urgent | 07/11/2016 | 08/10/2018 | Operations | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk | EGI |
GGUS Tickets Closed Last week |
Request id | Affected vo | Status | Priority | Date of creation | Last update | Type of problem | Subject | Scope |
---|---|---|---|---|---|---|---|---|
137723 | cms | solved | urgent | 14/10/2018 | 15/10/2018 | CMS_Data Transfers | PhEDEx component Agent Watchdog donw at T1_UK-RAL | WLCG |
137699 | dteam | verified | top priority | 11/10/2018 | 15/10/2018 | Monitoring | Test of RAL-LCG2 Alarm Ticket Handling | WLCG |
137675 | ops | verified | less urgent | 10/10/2018 | 11/10/2018 | Operations | [Rod Dashboard] Issue detected : org.nordugrid.ARC-CE-ARIS@arc-ce04.gridpp.rl.ac.uk | EGI |
137634 | cms | solved | urgent | 08/10/2018 | 10/10/2018 | CMS_Data Transfers | Transfers failing from T1_UK_RAL_Disk to METU | WLCG |
137619 | cms | solved | urgent | 07/10/2018 | 08/10/2018 | CMS_AAA WAN Access | T1_UK_RAL xrootd read failures | WLCG |
137434 | cms | closed | urgent | 26/09/2018 | 11/10/2018 | CMS_AAA WAN Access | xrootd host not reachable at T1_UK_RAL | WLCG |
137391 | atlas | solved | urgent | 25/09/2018 | 10/10/2018 | Network problem | UK RAL-LCG2 transfer errors with Communication error on send | WLCG |
137151 | t2k.org | closed | urgent | 12/09/2018 | 08/10/2018 | File Transfer | Cannot upload files to grid an register replica | EGI |
136757 | mice | closed | less urgent | 17/08/2018 | 09/10/2018 | Other | Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ? | EGI |
136028 | cms | closed | top priority | 10/07/2018 | 12/10/2018 | CMS_AAA WAN Access | Issues reading files at T1_UK_RAL_Disk | WLCG |
Availability Report |
Day | Atlas | Atlas-Echo | CMS | LHCB | Alice |
---|---|---|---|---|---|
2018-10-08 | 100 | 100 | 100 | 100 | 100 |
2018-10-09 | 100 | 100 | 99 | 100 | 100 |
2018-10-10 | 100 | 100 | 98 | 100 | 100 |
2018-10-11 | 100 | 100 | 98 | 100 | 100 |
2018-10-12 | 100 | 100 | 100 | 100 | 100 |
2018-10-13 | 100 | 100 | 100 | 100 | 100 |
2018-10-14 | 100 | 100 | 100 | 100 | 100 |
2018-10-15 | 100 | 100 | 100 | 100 | 100 |
Hammercloud Test Report |
Target Availability for each site is 97.0% | Red <90% | Orange <97% |
Day | Atlas HC | CMS HC | Comment |
---|---|---|---|
2018-08-01 | 100 | 98 | |
2018-08-02 | 100 | 99 | |
2018-08-03 | 100 | 99 | |
2018-08-04 | 100 | 99 | |
2018-08-05 | 100 | 98 | |
2018-08-06 | 100 | 98 | |
2018-08-07 | 100 | 96 | |
2018-08-08 | 100 | 96 |
Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud
Notes from Meeting. |