Tier1 Operations Report 2018-09-10

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 10th September 2018

Review of Issues during the week 27th August to the 10th September 2018.
Current operational status and issues
  • NTR
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss771 LHCb lhcbDst d1t0 Back in production RO
gdss732 LHCb lhcbDst d1t0 Back in production RO
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss747 Atlas atlasStripInput d1t0 Currently in intervention.
gdss738 LHCb lhcbDst d1t0 Currently in intervention.
Limits on concurrent batch system jobs.
  • GROUP_CMS_LIMIT = 4000
  • GROUP_ATLAS_LIMIT = 8000
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
25905 FTS UNSCHEDULED OUTAGE 24/8/18 14:30 24/8/18 18:45 - IPv6 outage impacting FTS services
25912 GOCDB UNSCHEDULED OUTAGE 25/8/18 18:50 25/8/18 19:46 - Ongoing IPv6 outage impacting GOCDB
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
136884 t2k.org in progress top priority 27/08/2018 29/08/2018 Data Management - generic lcg-cr not working for t2k vo EGI
136840 snoplus.snolab.ca in progress urgent 23/08/2018 04/09/2018 Other Cannot upload files to LFN from Storage node EGI
136757 mice in progress less urgent 17/08/2018 21/08/2018 Other Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ? EGI
136701 lhcb in progress very urgent 14/08/2018 05/09/2018 File Transfer background of transfer errors WLCG
136366 mice in progress less urgent 25/07/2018 20/08/2018 Local Batch System Remove MICE Queue from RAL T1 Batch EGI
136199 lhcb in progress very urgent 18/07/2018 07/08/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
136028 cms in progress top priority 10/07/2018 07/09/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG
124876 ops in progress less urgent 07/11/2016 23/07/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
136967 cms solved urgent 31/08/2018 04/09/2018 CMS_Data Transfers Phedex transfers failing from RAL to FNAL WLCG
136942 t2k.org solved less urgent 29/08/2018 06/09/2018 File Transfer Copying ONLINE_AND_NEARLINE files from RAL tape storage times out EGI
136914 mice solved less urgent 28/08/2018 30/08/2018 Storage Systems Lock-down of MICE Miscellaneous data on Castor EGI
136913 mice solved less urgent 28/08/2018 29/08/2018 Storage Systems Lock-down of MICE RAW data EGI
136896 lhcb verified urgent 27/08/2018 29/08/2018 File Access File unaccessible WLCG
136865 cms solved urgent 24/08/2018 28/08/2018 CMS_Facilities T1_UK_RAL squid failures and warns WLCG
136863 lhcb verified top priority 24/08/2018 28/08/2018 File Transfer RAL FTS failing all transfers WLCG
136859 cms solved top priority 23/08/2018 28/08/2018 CMS_Data Transfers RAL FTS server is down WLCG
136665 cms closed urgent 11/08/2018 04/09/2018 CMS_SAM tests T1_UK_RAL is down > 12h WLCG
136563 cms solved urgent 06/08/2018 28/08/2018 CMS_Data Transfers Possibly corrupted files at RAL WLCG
136358 cms closed urgent 25/07/2018 04/09/2018 CMS_Facilities T1_UK_RAL WN-xrootd-access failure WLCG

Availability Report

Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-08-27 100 100 100 100 100 100
2018-08-28 100 100 98 100 100 100
2018-08-29 100 100 100 100 100 100
2018-08-30 100 100 100 100 100 100
2018-08-31 100 100 100 100 100 100
2018-09-01 100 100 100 100 100 100
2018-09-02 100 100 99 100 100 100
2018-09-03 100 100 100 100 100 100
2018-09-04 100 100 99 100 100 100
2018-09-05 100 100 99 100 100 100
2018-09-06 92 100 92 100 98 87
2018-09-07 100 100 100 100 100 100
2018-09-08 100 100 100 100 100 100
2018-09-09 100 100 100 100 100 100
2018-09-10 100 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-08-13 0 0
2018-08-14 0 0
2018-08-15 0 0
2018-08-16 0 0
2018-08-17 76 60
2018-08-18 100 100
2018-08-19 100 100
2018-08-20 100 100

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-09-10&oldid=18430"