Difference between revisions of "Tier1 Operations Report 2017-04-19"
From GridPP Wiki
(→) |
|||
Line 10: | Line 10: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 12th to 19th April 2017. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 12th to 19th April 2017. | ||
|} | |} | ||
− | * Problems with the LHCb Castor instance | + | * Problems with the LHCb Castor instance continued until Wednesday 12th April when the SRM upgrade was reverted for LHCb. Since then LHCb Castor has performed much better and the backlog of work has largely been eliminated. |
− | + | * On Wednesday, 12th April a problem with certificate ownership on eight Atlas disk servers was identified. This was causing some SAM test failures. The incorrect certificate ownership was present on these servers for around 24 hours before the problem was identified and fixed. | |
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
<!-- *********************************************************** -----> | <!-- *********************************************************** -----> | ||
Line 22: | Line 22: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues | ||
|} | |} | ||
− | * | + | * gdss673 (LHCb-Tape) was removed from production on 5th April due to it having a double disk failure. It was returned to service on Thursday (13th April) after several disks were replaced. |
<!-- ***************************************************** -----> | <!-- ***************************************************** -----> | ||
Line 43: | Line 43: | ||
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues | | style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues | ||
|} | |} | ||
− | * | + | * None |
<!-- ***************End Ongoing Disk Server Issues**************** -----> | <!-- ***************End Ongoing Disk Server Issues**************** -----> | ||
<!-- ************************************************************* -----> | <!-- ************************************************************* -----> |
Revision as of 19:43, 18 April 2017
RAL Tier1 Operations Report for 19th April2017
Review of Issues during the week 12th to 19th April 2017. |
- Problems with the LHCb Castor instance continued until Wednesday 12th April when the SRM upgrade was reverted for LHCb. Since then LHCb Castor has performed much better and the backlog of work has largely been eliminated.
- On Wednesday, 12th April a problem with certificate ownership on eight Atlas disk servers was identified. This was causing some SAM test failures. The incorrect certificate ownership was present on these servers for around 24 hours before the problem was identified and fixed.
Resolved Disk Server Issues |
- gdss673 (LHCb-Tape) was removed from production on 5th April due to it having a double disk failure. It was returned to service on Thursday (13th April) after several disks were replaced.
Current operational status and issues |
- We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities but the level of failures is reduced as compared to a few weeks ago.
Ongoing Disk Server Issues |
- None
Limits on concurrent batch system jobs. |
- Atlas Pilot (Analysis) 1500
- CMS Multicore 550
- LHCb 500 (In response to ongoing problems with LHCb Castor.)
Notable Changes made since the last meeting. |
- Increased limit on number of CMS multicore jobs from 460 to 550 due to increased pledge for 2017.
- Out of Hours cover for the CEPH ECHO service is being piloted.
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
lcgwms04.gridpp.rl.ac.uk | SCHEDULED | OUTAGE | 12/04/2017 09:05 | 18/04/2017 12:00 | 6 days, 2 hours and 55 minutes | server migration |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
Pending - but not yet formally announced:
- Update Castor SRMs - CMS & GEN still to do. This is awaiting a full understanding of the problem seen with LHCb.
- Chiller replacement - work ongoing. (First chiller replacement completed, second in place).
- Merge AtlasScratchDisk into larger Atlas disk pool.
Listing by category:
- Castor:
- Update SRMs to new version, including updating to SL6.
- Bring some newer disk servers ('14 generation) into service, replacing some older ('12 generation) servers.
- Networking
- Enable first services on production network with IPv6 once addressing scheme agreed.
- Infrastructure:
- Two of the chillers supplying the air-conditioning for the R89 machine room are being replaced.
Entries in GOC DB starting since the last report. |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
lcgwms04.gridpp.rl.ac.uk | SCHEDULED | OUTAGE | 12/04/2017 09:05 | 18/04/2017 12:00 | 6 days, 2 hours and 55 minutes | server migration |
srm-lhcb.gridpp.rl.ac.uk | UNSCHEDULED | OUTAGE | 09/04/2017 12:00 | 10/04/2017 12:00 | 24 hours | Problems with LHCb transfers |
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
127617 | Green | Alarm | In Progress | 2017-04-09 | 2017-04-09 | LHCb | RAL srm-s are down for LHCb |
127612 | Green | Alarm | In Progress | 2017-04-08 | 2017-04-10 | LHCb | CEs at RAL not responding |
127611 | Green | Top Priority | In Progress | 2017-04-08 | 2017-04-11 | Alice | ALICE VOBOX network problem |
127598 | Green | Urgent | In Progress | 2017-04-07 | 2017-04-07 | CMS | UK XRootD Redirector |
127597 | Green | Urgent | In Progress | 2017-04-07 | 2017-04-10 | CMS | Check networking and xrootd RAL-CERN performance |
127388 | Green | Less urgent | In Progress | 2017-03-29 | 2017-04-03 | LHCb | [FATAL] Connection error for some file |
127240 | Green | Urgent | In Progress | 2017-03-21 | 2017-04-05 | CMS | Staging Test at UK_RAL for Run2 |
126905 | Green | Less Urgent | Waiting Reply | 2017-03-02 | 2017-04-03 | solid | finish commissioning cvmfs server for solidexperiment.org |
126184 | Amber | Less Urgent | In Progress | 2017-01-26 | 2017-02-07 | Atlas | Request of inputs for new sites monitoring |
124876 | Red | Less Urgent | On Hold | 2016-11-07 | 2017-01-01 | OPS | [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk |
117683 | Red | Less Urgent | On Hold | 2015-11-18 | 2017-03-02 | CASTOR at RAL not publishing GLUE 2. |
Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC ECHO = Atlas ECHO (Template 841);CMS HC = CMS HammerCloud
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | Atlas HC ECHO | CMS HC | Comment |
---|---|---|---|---|---|---|---|---|---|
12/04/17 | 100 | 100 | 71 | 98 | 54 | 100 | 100 | 97 | Atlas: Problem with certificate ownership on some disk servers; CMS - SRM test failure: Unable to issue PrepareToPut request to Castor; LHCb - Castor/SRM problem - including short (less than hour) outage to downgrade SRM. |
13/04/17 | 100 | 100 | 92 | 98 | 100 | 99 | 100 | N/A | Atlas - Tail end of credentials problem. |
14/04/17 | 100 | 100 | 98 | 99 | 83 | 100 | 100 | N/A | Atlas - Single SRM test failure; LHCb - high load caused some test failures. |
15/04/17 | 100 | 100 | 98 | 100 | 100 | 99 | 100 | N/A | Some data missing from SAM results |
16/04/17 | 100 | 100 | 100 | 98 | 100 | 100 | 100 | N/A | Some data missing from SAM results |
17/04/17 | 100 | 100 | 100 | 99 | 100 | 87 | 100 | N/A | Some data missing from SAM results |
18/04/17 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | N/A | Some data missing from SAM results |
Notes from Meeting. |
- None yet