Difference between revisions of "Tier1 Operations Report 2017-04-26"

From GridPP Wiki
Jump to: navigation, search
(→‎)
(→‎)
Line 66: Line 66:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
|}
 
|}
−
* LHCb Castor SRM downgraded to version 2.11.
+
* Atlas Pilot (Analysis) job limit of 1500 removed.
−
* Site Argus sytems put behind load balancer.
+
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->

Revision as of 18:54, 25 April 2017

RAL Tier1 Operations Report for 26th April2017

Review of Issues during the week 19th to 26th April 2017.
  • Problems with the LHCb Castor instance continued until Wednesday 12th April when the SRM upgrade was reverted for LHCb. Since then LHCb Castor has performed much better and the backlog of work has largely been eliminated.
  • On Wednesday, 12th April a problem with certificate ownership on eight Atlas disk servers was identified. This was causing some SAM test failures. The incorrect certificate ownership was present on these servers for around 24 hours before the problem was identified and fixed.
Resolved Disk Server Issues
  • GDSS722 (AtlasDataDisk - D1T0) was taken out of service on Tuesday 18th April for memory errors to be investigated. Following a swap round and re-seating of its memory it was returned to service the following day (Wednesday 19th April).
  • GDSS737 (LHCbDst - D1T0) was taken out of service on Monday, 24th April after two disk drives had failed. It returned to service, intially read-only, yesterday (25th April).
Current operational status and issues
  • We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities but the level of failures is reduced as compared to a few weeks ago.
Ongoing Disk Server Issues
  • GDSS729 (CMSDisk - D1tT0) was taken out of service yesterday (18th April) after it became unresponsive. Investigations are ongoing.
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • Atlas Pilot (Analysis) job limit of 1500 removed.
Declared in the GOC DB

None

Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Pending - but not yet formally announced:

  • Update Castor SRMs - CMS & GEN still to do. Problems seen with the SRM update mean these will probably wait until Castor 2.1.16 is rolled out.
  • Chiller replacement - work ongoing. (First chiller replacement completed, second in place).
  • Merge AtlasScratchDisk into larger Atlas disk pool.

Listing by category:

  • Castor:
    • Update SRMs to new version, including updating to SL6.
    • Update to version 2.1.16
  • Networking
    • Enable first services on production network with IPv6 now that the addressing scheme has been agreed.
  • Infrastructure:
    • Two of the chillers supplying the air-conditioning for the R89 machine room are being replaced.
Entries in GOC DB starting since the last report.
  • None
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
127720 Green Urgent In Progress 2017-04-13 2017-04-13 Atlas RAL-LCG2 frontier squid has been down in the last few days
127617 Green Alarm In Progress 2017-04-09 2017-04-10 LHCb RAL srm-s are down for LHCb
127612 Green Alarm In Progress 2017-04-08 2017-04-10 LHCb CEs at RAL not responding
127611 Green Top Priority In Progress 2017-04-08 2017-04-13 Alice ALICE VOBOX network problem
127598 Green Urgent In Progress 2017-04-07 2017-04-07 CMS UK XRootD Redirector
127597 Green Urgent In Progress 2017-04-07 2017-04-10 CMS Check networking and xrootd RAL-CERN performance
127240 Green Urgent In Progress 2017-03-21 2017-04-05 CMS Staging Test at UK_RAL for Run2
126905 Green Less Urgent In progress 2017-03-02 2017-04-18 solid finish commissioning cvmfs server for solidexperiment.org
126184 Amber Less Urgent In Progress 2017-01-26 2017-02-07 Atlas Request of inputs for new sites monitoring
124876 Red Less Urgent On Hold 2016-11-07 2017-01-01 OPS [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk
117683 Red Less Urgent On Hold 2015-11-18 2017-03-02 CASTOR at RAL not publishing GLUE 2.
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC ECHO = Atlas ECHO (Template 841);CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC Atlas HC ECHO CMS HC Comment
19/04/17 100 100 98 93 100 97 100 99 Atlas & CMS: SRM test failures.
20/04/17 100 100 94 98 84 97 100 100 Atlas, CMS, & LHCb: SRM Test failures;
21/04/17 100 100 87 100 92 100 96 100 Atlas & LHCb: SRM Test failures
22/04/17 100 100 84 96 100 47 98 100 Atlas & CMS: SRM errors
23/04/17 100 98 90 97 100 0 100 100 Alice: Missing data; Atlas - pool full; CMS - SRM errors
24/04/17 100 100 100 95 79 24 100 100 SRM test failures for both VOs.
25/04/17 100 100 100 100 100 100 100 100 SRM test failures for both VOs.
Notes from Meeting.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2017-04-26&oldid=14162"