Difference between revisions of "Tier1 Operations Report 2017-05-03"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
Line 226: Line 226:
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! Atlas HC ECHO !! CMS HC !! Comment
 
! Day !! OPS !! Alice !! Atlas !! CMS !! LHCb !! Atlas HC !! Atlas HC ECHO !! CMS HC !! Comment
 
|-
 
|-
−
| 19/04/17 || 100 || 100 || style="background-color: lightgrey;" | 98 || style="background-color: lightgrey;" | 93 || 100 || 97 || 100 || 99 || Atlas & CMS: SRM test failures.
+
| 26/04/17 || 100 || 100 || style="background-color: lightgrey;" | 90 || style="background-color: lightgrey;" | 82 || style="background-color: lightgrey;" | 92 || 22 || 92 || 99 || All SRM test failures: Atlas - unknown errors. CMS - SRM failures; LHCb - SRM test failures.
 
|-
 
|-
−
| 20/04/17 || 100 || 100 || style="background-color: lightgrey;" | 94 || style="background-color: lightgrey;" | 98 || style="background-color: lightgrey;" | 84 || 97 || 100 || 100 || Atlas, CMS, & LHCb: SRM Test failures;
+
| 27/04/17 || 100 || 100 || style="background-color: lightgrey;" | 85 || style="background-color: lightgrey;" | 86 || style="background-color: lightgrey;" | 92 || 51 || 99 || 100 || All SRM test failures: Atlas - unknown errors. CMS - SRM failures and batch job failures. LHCb - SRM test failures.
 
|-
 
|-
−
| 21/04/17 || 100 || 100 || style="background-color: lightgrey;" | 87 || 100 || style="background-color: lightgrey;" | 92 || 100 || 96 || 100 || Atlas & LHCb: SRM Test failures
+
| 28/04/17 || 100 || 100 || style="background-color: lightgrey;" | 96 || style="background-color: lightgrey;" | 81 || style="background-color: lightgrey;" | 97 || 93 || 100 || N/A || All SRM test failures: Atlas - unknown errors. CMS: SRM test failures. LHCb - SRM test failures.
 
|-
 
|-
−
| 22/04/17 || 100 || 100 || style="background-color: lightgrey;" | 84 || style="background-color: lightgrey;" | 96 || 100 || 47 || 98 || 100 || Atlas & CMS: SRM errors
+
| 29/04/17 || 100 || 100 || style="background-color: lightgrey;" | 85 || style="background-color: lightgrey;" | 79 || style="background-color: lightgrey;" | 99 || 99 || 100 || N/A || All SRM test failures: Atlas - unknown errors. CMS: SRM test failures. LHCb - SRM test failure.
 
|-
 
|-
−
| 23/04/17 || 100 || style="background-color: lightgrey;" | 98 || style="background-color: lightgrey;" | 90 || style="background-color: lightgrey;" | 97 || 100 || 0 || 100 || 100 || Alice: Missing data;  Atlas - pool full; CMS - SRM errors
+
| 30/04/17 || 100 || 100 || style="background-color: lightgrey;" | 6 || style="background-color: lightgrey;" | 63 || 100 || 97 || 100 || 91 || Atlas - disk full. CMS: SRM test failures.
 
|-
 
|-
−
| 24/04/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 95 || style="background-color: lightgrey;" | 79 || 24 || 100 || 100 || SRM test failures for both VOs.
+
| 01/05/17 || 100 || 100 || style="background-color: lightgrey;" | 0 || style="background-color: lightgrey;" | 68 || style="background-color: lightgrey;" | 96 || 97 || 100 || 100 || Atlas - disk full. CMS: SRM test failures. LHCb - SRM test failures.
 
|-
 
|-
−
| 25/04/17 || 100 || 100 || 100 || style="background-color: lightgrey;" | 85 || style="background-color: lightgrey;" | 71 || 100 || 99 || 99 || SRM test failures for both VOs.
+
| 02/05/17 || 100 || 100 || style="background-color: lightgrey;" | 58 || style="background-color: lightgrey;" | 74 || style="background-color: lightgrey;" | 96 || 98 || 98 || 99 || Atlas: SRM test failures as DataDisk full; CMS: CE test failures owing to glexec (argus) problem;  LHCb: Single SRM test failure.
−
 
+
−
|-
+
−
| 26/04/17 || 100 || 100 || 90 || 82 || 92 || 100 || 100 || 100 || All SRM test failures: Atlas - unknown errors. CMS - SRM failures; LHCb - SRM test failures.
+
−
|-
+
−
| 27/04/17 || 100 || 100 || 85 || 86 || 92 || 100 || 100 || 100 || All SRM test failures: Atlas - unknown errors. CMS - SRM failures and batch job failures. LHCb - SRM test failures.
+
−
|-
+
−
| 28/04/17 || 100 || 100 || 96 || 81 || 97 || 100 || 100 || 100 || All SRM test failures: Atlas - unknown errors. CMS: SRM test failures. LHCb - SRM test failures.
+
−
|-
+
−
| 29/04/17 || 100 || 100 || 85 || 79 || 99 || 100 || 100 || 100 || All SRM test failures: Atlas - unknown errors. CMS: SRM test failures. LHCb - SRM test failure.
+
−
|-
+
−
| 30/04/17 || 100 || 100 || 6 || 63 || 100 || 100 || 100 || 100 || Atlas - disk full. CMS: SRM test failures.
+
−
|-
+
−
| 01/05/17 || 100 || 100 || 0 || 68 || 96 || 100 || 100 || 100 || Atlas - disk full. CMS: SRM test failures. LHCb - SRM test failures.
+
−
|-
+
−
| 02/05/17 || 100 || 100 || 58 || 174 || 96 || 100 || 100 || 100 || Atlas: SRM test failures as DataDisk full; CMS: CE test failures owing to glexec (argus) problem;  LHCb: Single SRM test failure.
+
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->

Revision as of 07:56, 3 May 2017

RAL Tier1 Operations Report for 3rd May 2017

Review of Issues during the week 26th April to 3rd May 2017.
  • Following the reversion of the LHCb Castor SRMs on Wednesday 12th April Castor for LHCb has worked much better although some significant problems have remained. The total load on LHCb Castor has had to be managed by LHCb and the there have been times when the SRMs clogged up requiring manual intervention to restart things. Nevertheless the appreciable backlog of LHCb stripping and merging jobs that had built up has been largely eliminated.
  • There was a problem with Castor for CMS overnight last night (Tuesday/Wednesday 25/26 April) when the database reported blocking sessions. This fixed itself after a couple of hours.
  • There was a problems for Atlas (as shown in the Hammercloud test results below) over the weekend when AtalsScratchDisk filled up. This space is managed by Atlas.
  • We are also seeing a non-negligible failure rate for CE SAM tests for CMS that is not yet understood.
Resolved Disk Server Issues
  • GDSS722 (AtlasDataDisk - D1T0) was taken out of service on Tuesday 18th April for memory errors to be investigated. Following a swap round and re-seating of its memory it was returned to service the following day (Wednesday 19th April).
  • GDSS737 (LHCbDst - D1T0) was taken out of service on Monday, 24th April after two disk drives had failed. It returned to service, intially read-only, yesterday (25th April).
Current operational status and issues
  • We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities.
Ongoing Disk Server Issues
  • GDSS729 (CMSDisk - D1T0) was taken out of service yesterday (18th April) after it became unresponsive. Investigations are ongoing.
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • Atlas Pilot (Analysis) job limit of 1500 removed.
  • Changed Castor non-ATLAS 'default' service classes to point to the D0T1 endpoints to better handle transfers that do not specify a service class.
Declared in the GOC DB

None

Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Pending - but not yet formally announced:

  • Update Castor SRMs - CMS & GEN still to do. Problems seen with the SRM update mean these will probably wait until Castor 2.1.16 is rolled out.
  • Merge AtlasScratchDisk into larger Atlas disk pool.

Listing by category:

  • Castor:
    • Update SRMs to new version, including updating to SL6.
    • Update to version 2.1.16
  • Networking
    • Enable first services on production network with IPv6 now that the addressing scheme has been agreed. (Perfsonar already working over IPv6).
Entries in GOC DB starting since the last report.
  • None
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
127916 Green Alarm In Progress 2017-04-25 2017-04-25 LHCb RAL srm-s are down again for LHCb
127612 Green Alarm In Progress 2017-04-08 2017-04-10 LHCb CEs at RAL not responding
127598 Green Urgent In Progress 2017-04-07 2017-04-07 CMS UK XRootD Redirector
127597 Green Urgent In Progress 2017-04-07 2017-04-10 CMS Check networking and xrootd RAL-CERN performance
127388 Green Less Urgent In Progress 2017-03-29 2017-04-20 LHCb [FATAL] Connection error for some file
127240 Green Urgent In Progress 2017-03-21 2017-04-05 CMS Staging Test at UK_RAL for Run2
126905 Green Less Urgent Waiting Reply 2017-03-02 2017-04-21 solid finish commissioning cvmfs server for solidexperiment.org
124876 Red Less Urgent On Hold 2016-11-07 2017-01-01 OPS [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
117683 Red Less Urgent On Hold 2015-11-18 2017-03-02 CASTOR at RAL not publishing GLUE 2.
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC ECHO = Atlas ECHO (Template 841);CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC Atlas HC ECHO CMS HC Comment
26/04/17 100 100 90 82 92 22 92 99 All SRM test failures: Atlas - unknown errors. CMS - SRM failures; LHCb - SRM test failures.
27/04/17 100 100 85 86 92 51 99 100 All SRM test failures: Atlas - unknown errors. CMS - SRM failures and batch job failures. LHCb - SRM test failures.
28/04/17 100 100 96 81 97 93 100 N/A All SRM test failures: Atlas - unknown errors. CMS: SRM test failures. LHCb - SRM test failures.
29/04/17 100 100 85 79 99 99 100 N/A All SRM test failures: Atlas - unknown errors. CMS: SRM test failures. LHCb - SRM test failure.
30/04/17 100 100 6 63 100 97 100 91 Atlas - disk full. CMS: SRM test failures.
01/05/17 100 100 0 68 96 97 100 100 Atlas - disk full. CMS: SRM test failures. LHCb - SRM test failures.
02/05/17 100 100 58 74 96 98 98 99 Atlas: SRM test failures as DataDisk full; CMS: CE test failures owing to glexec (argus) problem; LHCb: Single SRM test failure.
Notes from Meeting.
  • None yet
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2017-05-03&oldid=14260"