Difference between revisions of "Tier1 Operations Report 2017-04-19"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
 
(14 intermediate revisions by 2 users not shown)
Line 10: Line 10:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 12th to 19th  April 2017.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 12th to 19th  April 2017.
 
|}
 
|}
−
* Problems with the LHCb Castor instance have continued this last week. Under sustained load the service (the SRMs) fail. We are working with LHCb on the problem and are currently running with a reduced job rate of the LHCb merging jobs with are a particular cause of load on Castor. I note that a GGUS alarm ticket was received from LHCb about this problem. Note added just after meeting. As agreed we will revert the SRM update that was made on 23rd March.
+
* Problems with the LHCb Castor instance continued until Wednesday 12th April when the SRM upgrade was reverted for LHCb. Since then LHCb Castor has performed much better and the backlog of work has largely been eliminated.
−
* We have also seen problems with the Atlas Castor instance (see availability report below). This Castor instance is running the same Castor SRM version as LHCb.
+
* On Wednesday, 12th April a problem with certificate ownership on eight Atlas disk servers was identified. This was causing some SAM test failures. The incorrect certificate ownership was present on these servers for around 24 hours before the problem was identified and fixed.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 22: Line 22:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
−
* None
+
* gdss673 (LHCb-Tape) was removed from production on 5th April due to it having a double disk failure. It was returned to service on Thursday (13th April) after several disks were replaced.
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
  
Line 43: Line 43:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues
 
|}
 
|}
−
* gdss673 (LHCb-Tape) was removed from production last Wednesday (5th April) due to it having a double disk failure. Fixing this is ongoing.
+
* GDSS722 (AtlasDataDisk - D1tT0) was taken out of service yesterday (18th April) for memory errors to be investigated.
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ************************************************************* ----->
 
<!-- ************************************************************* ----->
Line 56: Line 56:
 
* Atlas Pilot (Analysis) 1500
 
* Atlas Pilot (Analysis) 1500
 
* CMS Multicore 550
 
* CMS Multicore 550
−
* LHCb 500 (In response to ongoing problems with LHCb Castor.)
 
 
<!-- ******************End Limits On Batch System Jobs***************** ----->
 
<!-- ******************End Limits On Batch System Jobs***************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 67: Line 66:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
|}
 
|}
−
* Increased limit on number of CMS multicore jobs from 460 to 550 due to increased pledge for 2017.
+
* LHCb Castor SRM downgraded to version 2.11.
−
* Out of Hours cover for the CEPH ECHO service is being piloted.
+
* Site Argus sytems put behind load balancer.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 79: Line 78:
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB
 
|}
 
|}
−
{| border=1 align=center
+
None
−
|- bgcolor="#7c8aaf"
+
−
! Service
+
−
! Scheduled?
+
−
! Outage/At Risk
+
−
! Start
+
−
! End
+
−
! Duration
+
−
! Reason
+
−
|-
+
−
| lcgwms04.gridpp.rl.ac.uk
+
−
| SCHEDULED
+
−
| OUTAGE
+
−
| 12/04/2017 09:05
+
−
| 18/04/2017 12:00
+
−
| 6 days, 2 hours and 55 minutes
+
−
| server migration
+
−
|}
+
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 111: Line 93:
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
<!-- ******* still to be formally scheduled and/or announced ******* ----->
 
'''Pending - but not yet formally announced:'''
 
'''Pending - but not yet formally announced:'''
−
* Update Castor SRMs - CMS & GEN still to do. This is awaiting a full understanding of the problem seen with LHCb.
+
* Update Castor SRMs - CMS & GEN still to do. Problems seen with the SRM update mean these will probably wait until Castor 2.1.16 is rolled out.
 
* Chiller replacement - work ongoing. (First chiller replacement completed, second in place).
 
* Chiller replacement - work ongoing. (First chiller replacement completed, second in place).
 
* Merge AtlasScratchDisk into larger Atlas disk pool.
 
* Merge AtlasScratchDisk into larger Atlas disk pool.
Line 117: Line 99:
 
* Castor:  
 
* Castor:  
 
** Update SRMs to new version, including updating to SL6.
 
** Update SRMs to new version, including updating to SL6.
−
** Bring some newer disk servers ('14 generation) into service, replacing some older ('12 generation) servers.
+
** Update to version 2.1.16
 
* Networking
 
* Networking
−
** Enable first services on production network with IPv6 once addressing scheme agreed.
+
** Enable first services on production network with IPv6 now that the addressing scheme has been agreed.
 
* Infrastructure:
 
* Infrastructure:
 
** Two of the chillers supplying the air-conditioning for the R89 machine room are being replaced.
 
** Two of the chillers supplying the air-conditioning for the R89 machine room are being replaced.
Line 141: Line 123:
 
! Duration
 
! Duration
 
! Reason
 
! Reason
 +
|-
 +
|srm-lhcb.gridpp.rl.ac.uk
 +
| UNSCHEDULED
 +
| OUTAGE
 +
| 12/04/2017 13:45
 +
| 12/04/2017 15:49
 +
| 2 hours and 4 minutes
 +
| LHCb SRMs being downgraded in response to recent problems
 
|-
 
|-
 
| lcgwms04.gridpp.rl.ac.uk
 
| lcgwms04.gridpp.rl.ac.uk
Line 146: Line 136:
 
| OUTAGE
 
| OUTAGE
 
| 12/04/2017 09:05
 
| 12/04/2017 09:05
−
| 18/04/2017 12:00
+
| 12/04/2017 10:57
−
| 6 days, 2 hours and 55 minutes
+
| 1 hour and 52 minutes
 
| server migration  
 
| server migration  
−
|-
 
−
| srm-lhcb.gridpp.rl.ac.uk
 
−
| UNSCHEDULED
 
−
| OUTAGE
 
−
| 09/04/2017 12:00
 
−
| 10/04/2017 12:00
 
−
| 24 hours
 
−
| Problems with LHCb transfers
 
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 172: Line 154:
 
|-style="background:#b7f1ce"
 
|-style="background:#b7f1ce"
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
 
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject
−
 
+
|-
 +
| 127720
 +
| Green
 +
| Urgent
 +
| In Progress
 +
| 2017-04-13
 +
| 2017-04-13
 +
| Atlas
 +
| RAL-LCG2 frontier squid has been down in the last few days
 
|-
 
|-
 
| 127617
 
| 127617
Line 179: Line 169:
 
| In Progress
 
| In Progress
 
| 2017-04-09
 
| 2017-04-09
−
| 2017-04-09
+
| 2017-04-10
 
| LHCb
 
| LHCb
 
| RAL srm-s are down for LHCb
 
| RAL srm-s are down for LHCb
Line 197: Line 187:
 
| In Progress
 
| In Progress
 
| 2017-04-08
 
| 2017-04-08
−
| 2017-04-11
+
| 2017-04-13
 
| Alice
 
| Alice
 
| ALICE VOBOX network problem
 
| ALICE VOBOX network problem
Line 218: Line 208:
 
| CMS
 
| CMS
 
| Check networking and xrootd RAL-CERN performance
 
| Check networking and xrootd RAL-CERN performance
−
|-
 
−
| 127388
 
−
| Green
 
−
| Less urgent
 
−
| In Progress
 
−
| 2017-03-29
 
−
| 2017-04-03
 
−
| LHCb
 
−
| [FATAL] Connection error for some file
 
 
|-
 
|-
 
| 127240
 
| 127240
Line 240: Line 221:
 
| Green
 
| Green
 
| Less Urgent
 
| Less Urgent
−
| Waiting Reply
+
| In progress
 
| 2017-03-02
 
| 2017-03-02
−
| 2017-04-03
+
| 2017-04-18
 
| solid
 
| solid
 
| finish commissioning cvmfs server for solidexperiment.org
 
| finish commissioning cvmfs server for solidexperiment.org
Line 289: Line 270:
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! Atlas HC ECHO !! CMS HC !! Comment
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! Atlas HC ECHO !! CMS HC !! Comment
 
|-
 
|-
−
| 05/04/17 || 100 || 100 || 100 || 100 || style="background-color: lightgrey;" | 88 || 100 || 100 || N/A || SRM Test failures.
+
| 12/04/17 || 100 || 100 || style="background-color: lightgrey;" | 71 || style="background-color: lightgrey;" | 98 || style="background-color: lightgrey;" | 54 || 100 || 100 || 97 || Atlas: Problem with certificate ownership on some disk servers; CMS - SRM test failure: Unable to issue PrepareToPut request to Castor; LHCb - Castor/SRM problem - including short (less than hour) outage to downgrade SRM.
−
|-
+
−
| 06/04/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 99 || style="background-color: lightgrey;" | 84 || 100 || 100 || 100 || SRM test failures for both CMS and LHCb.
+
−
|-
+
−
| 07/04/17 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || 100 ||
+
−
|-
+
−
| 08/04/17 || 100 || style="background-color: lightgrey;" | 96 || style="background-color: lightgrey;" | 88 || style="background-color: lightgrey;" | 75 || style="background-color: lightgrey;" | 94 || 100 || 100 || 100 || A hypervisor failure led to problems for one of the CEs and argus.
+
−
|-
+
−
| 09/04/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 45 || style="background-color: lightgrey;" | 88 || 100 || 100 || 100 || CMS: Problem with CMS Castor (transfer manager problems); LHCb - SRM test failures.
+
−
|-
+
−
| 10/04/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 60 || 100 || 100 || 99 || 99 || CMS: Ongoing problem with CMS Castor (above) fixed during morning.
+
−
|-
+
−
| 11/04/17 || 100 || 100 || style="background-color: lightgrey;" | 58 || style="background-color: lightgrey;" | 93 || style="background-color: lightgrey;" | 80 || 98 || 98 || 99 || All three are SRM test failures caused by Castor problems.
+
−
 
+
−
 
+
−
|-
+
−
| 12/04/17 || 100 || 100 || 71 || 98 || 54 || 100 || 100 || 100 || Atlas: Problem with certificate ownership on some disk servers; CMS - SRM test failure: Unable to issue PrepareToPut request to Castor; LHCb - Castor/SRM problem - including short (less than hour) outage to downgrade SRM.
+
 
|-
 
|-
−
| 13/04/17 || 100 || 100 || 92 || 98 || 100 || 100 || 100 || 100 || Atlas - Tail end of credentials problem.  
+
| 13/04/17 || 100 || 100 || style="background-color: lightgrey;" | 92 || style="background-color: lightgrey;" | 98 || 100 || 99 || 100 || N/A || Atlas - Tail end of credentials problem.  
 
|-
 
|-
−
| 14/04/17 || 100 || 100 || 98 || 99 || 83 || 100 || 100 || 100 || Atlas - Single SRM test failure; LHCb - high load caused some test failures.
+
| 14/04/17 || 100 || 100 || style="background-color: lightgrey;" | 98 || style="background-color: lightgrey;" | 99 || style="background-color: lightgrey;" | 83 || 100 || 100 || N/A || Atlas - Single SRM test failure; LHCb - high load caused some test failures.
 
|-
 
|-
−
| 15/04/17 || 100 || 100 || 98 || 100 || 100 || 100 || 100 || 100 || Some data missing from SAM results
+
| 15/04/17 || 100 || 100 || style="background-color: lightgrey;" | 98 || 100 || 100 || 99 || 100 || N/A || Some data missing from SAM results
 
|-
 
|-
−
| 16/04/17 || 100 || 100 || 100 || 98 || 100 || 100 || 100 || 100 || Some data missing from SAM results
+
| 16/04/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 98 || 100 || 100 || 100 || N/A || Some data missing from SAM results
 
|-
 
|-
−
| 17/04/17 || 100 || 100 || 100 || 99 || 100 || 100 || 100 || 100 || Some data missing from SAM results
+
| 17/04/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 99 || 100 || 87 || 100 || 100 || Some data missing from SAM results
 
|-
 
|-
−
| 18/04/17 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || Some data missing from SAM results
+
| 18/04/17 || 100 || 100 || style="background-color: lightgrey;" | 99 ||  style="background-color: lightgrey;" | 92 || 100 || 100 || 100 || N/A || Some data missing from SAM results
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
Line 329: Line 294:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
|}
 
|}
−
* None yet
+
* The IPv6 addressing scheme for the tier1 has been agreed and is in place. We hope to start offering services via IPv6 in the near future.
 +
* On the 19th April, Atlas reached a milestone of having 1PB of data in ECHO

Latest revision as of 09:57, 23 April 2017

RAL Tier1 Operations Report for 19th April2017

Review of Issues during the week 12th to 19th April 2017.
  • Problems with the LHCb Castor instance continued until Wednesday 12th April when the SRM upgrade was reverted for LHCb. Since then LHCb Castor has performed much better and the backlog of work has largely been eliminated.
  • On Wednesday, 12th April a problem with certificate ownership on eight Atlas disk servers was identified. This was causing some SAM test failures. The incorrect certificate ownership was present on these servers for around 24 hours before the problem was identified and fixed.
Resolved Disk Server Issues
  • gdss673 (LHCb-Tape) was removed from production on 5th April due to it having a double disk failure. It was returned to service on Thursday (13th April) after several disks were replaced.
Current operational status and issues
  • We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities but the level of failures is reduced as compared to a few weeks ago.
Ongoing Disk Server Issues
  • GDSS722 (AtlasDataDisk - D1tT0) was taken out of service yesterday (18th April) for memory errors to be investigated.
Limits on concurrent batch system jobs.
  • Atlas Pilot (Analysis) 1500
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • LHCb Castor SRM downgraded to version 2.11.
  • Site Argus sytems put behind load balancer.
Declared in the GOC DB

None

Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Pending - but not yet formally announced:

  • Update Castor SRMs - CMS & GEN still to do. Problems seen with the SRM update mean these will probably wait until Castor 2.1.16 is rolled out.
  • Chiller replacement - work ongoing. (First chiller replacement completed, second in place).
  • Merge AtlasScratchDisk into larger Atlas disk pool.

Listing by category:

  • Castor:
    • Update SRMs to new version, including updating to SL6.
    • Update to version 2.1.16
  • Networking
    • Enable first services on production network with IPv6 now that the addressing scheme has been agreed.
  • Infrastructure:
    • Two of the chillers supplying the air-conditioning for the R89 machine room are being replaced.
Entries in GOC DB starting since the last report.
Service Scheduled? Outage/At Risk Start End Duration Reason
srm-lhcb.gridpp.rl.ac.uk UNSCHEDULED OUTAGE 12/04/2017 13:45 12/04/2017 15:49 2 hours and 4 minutes LHCb SRMs being downgraded in response to recent problems
lcgwms04.gridpp.rl.ac.uk SCHEDULED OUTAGE 12/04/2017 09:05 12/04/2017 10:57 1 hour and 52 minutes server migration
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
127720 Green Urgent In Progress 2017-04-13 2017-04-13 Atlas RAL-LCG2 frontier squid has been down in the last few days
127617 Green Alarm In Progress 2017-04-09 2017-04-10 LHCb RAL srm-s are down for LHCb
127612 Green Alarm In Progress 2017-04-08 2017-04-10 LHCb CEs at RAL not responding
127611 Green Top Priority In Progress 2017-04-08 2017-04-13 Alice ALICE VOBOX network problem
127598 Green Urgent In Progress 2017-04-07 2017-04-07 CMS UK XRootD Redirector
127597 Green Urgent In Progress 2017-04-07 2017-04-10 CMS Check networking and xrootd RAL-CERN performance
127240 Green Urgent In Progress 2017-03-21 2017-04-05 CMS Staging Test at UK_RAL for Run2
126905 Green Less Urgent In progress 2017-03-02 2017-04-18 solid finish commissioning cvmfs server for solidexperiment.org
126184 Amber Less Urgent In Progress 2017-01-26 2017-02-07 Atlas Request of inputs for new sites monitoring
124876 Red Less Urgent On Hold 2016-11-07 2017-01-01 OPS [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
117683 Red Less Urgent On Hold 2015-11-18 2017-03-02 CASTOR at RAL not publishing GLUE 2.
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC ECHO = Atlas ECHO (Template 841);CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC Atlas HC ECHO CMS HC Comment
12/04/17 100 100 71 98 54 100 100 97 Atlas: Problem with certificate ownership on some disk servers; CMS - SRM test failure: Unable to issue PrepareToPut request to Castor; LHCb - Castor/SRM problem - including short (less than hour) outage to downgrade SRM.
13/04/17 100 100 92 98 100 99 100 N/A Atlas - Tail end of credentials problem.
14/04/17 100 100 98 99 83 100 100 N/A Atlas - Single SRM test failure; LHCb - high load caused some test failures.
15/04/17 100 100 98 100 100 99 100 N/A Some data missing from SAM results
16/04/17 100 100 100 98 100 100 100 N/A Some data missing from SAM results
17/04/17 100 100 100 99 100 87 100 100 Some data missing from SAM results
18/04/17 100 100 99 92 100 100 100 N/A Some data missing from SAM results
Notes from Meeting.
  • The IPv6 addressing scheme for the tier1 has been agreed and is in place. We hope to start offering services via IPv6 in the near future.
  • On the 19th April, Atlas reached a milestone of having 1PB of data in ECHO
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2017-04-19&oldid=14132"