Tier1 Operations Report 2018-07-02

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 2nd July 2018

Review of Issues during the week 25th June to the 2nd July 2018.
  • Tier-1 are currently undertaking a campaign of better keep a track of our smaller VOs and trying to provide a more proactive approach to monitoring VOs usage (and where required, intervention). Among a number of methods we are using a combination of InfluxDB, Telegraf and Grafana we are now monitoring and alerting on weekly usage of all the smaller VOs.
Current operational status and issues
  • None.
Resolved Castor Disk Server Issues
  • None
Ongoing Castor Disk Server Issues
  • None.
Limits on concurrent batch system jobs.
  • CMS Multicore 550
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
  • None
Declared in the GOC DB
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Networking
    • Extend the number of services on the production network with IPv6 dual stack. (Done for Perfsonar, FTS3, all squids and the CVMFS Stratum-1 servers).
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
  • Infrastructure
    • There is currently a scheduled three day period (July 24th - July 26th), where RAL Tier-1 will be undertaking server room circuit breaker testing. Although it is believed that these tests should not effect our services, such is the nature of these tests all Tier-1 production services should be considered "At Risk" during this time. All VOs are asked to give consideration with respect to possible unexpected outage to any high priority/critical jobs they may wish to run during this period.

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135901 ops in progress less urgent 03/07/2018 03/07/2018 Operations [Rod Dashboard] Issue detected : org.bdii.GLUE2-Validate@site-bdii.gridpp.rl.ac.uk EGI
135822 cms in progress very urgent 26/06/2018 29/06/2018 CMS_Central Workflows File Read Problems for Production at T1_UK_RAL WLCG
134685 dteam in progress less urgent 23/04/2018 03/07/2018 Middleware please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7 EGI
124876 ops reopened less urgent 07/11/2016 28/06/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
135723 lhcb solved top priority 19/06/2018 28/06/2018 File Transfer lcgfts3 FTS server fails all transfers WLCG
135661 atlas closed less urgent 14/06/2018 28/06/2018 Databases RAL-LCG2: ATLAS RAL Frontier server down WLCG
135455 cms solved less urgent 31/05/2018 25/06/2018 File Transfer Checksum verification at RAL EGI
135367 snoplus.snolab.ca closed less urgent 28/05/2018 27/06/2018 Other Lost access to srm-snoplus.gridpp.rl.ac.uk EGI
135342 ops verified less urgent 27/05/2018 02/07/2018 Operations [Rod Dashboard] Issue detected : egi.eu.lowAvailability-/RAL-LCG2@RAL-LCG2_Availability EGI
135308 mice closed top priority 24/05/2018 27/06/2018 Information System Can't send data to RAL Castor EGI
135293 ops verified less urgent 23/05/2018 02/07/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI

Availability Report

Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-06-25 98 98 99 100 100 100
2018-06-26 100 100 98 100 100 100
2018-06-27 100 100 100 100 100 100
2018-06-28 100 100 100 100 100 100
2018-06-29 100 100 99 100 100 100
2018-06-30 100 100 100 100 100 100
2018-07-01 100 100 100 100 100 100
2018-07-02 100 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018/06/25 100 96
2018/06/26 0 100 0 : No test run in time bin
2018/06/27 0 58 0 : No test run in time bin
2018/06/28 0 100 0 : No test run in time bin
2018/06/29 0 92 0 : No test run in time bin
2018/06/30 100 98
2018/06/01 99 100

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
  • MICE
    • Will create a (GGUS) ticket to record the Castor file permission changes
  • SKA
    • There is an ongoing permissions problem that causes FTS file transfers to Echo to fail to overwrite files.
    • Need to resolve the access to various Rucio machines for the SKA contact in this area.
    • Question about FTS transfers appearing as either belonging to the SKA VO or to the individual submitter.
    • A couple of batch worker nodes are reserved for the SKA multi-core jobs (i.e. for those requiring a large number of cores).
  • Dune
    • Job Submissions. These are OK through ARC-CE01, ARC-CE02 and ARC-CE04. Awaiting confirmation that job submission via ARC-CE03 is now OK.
    • Dune are carrying out data transfer tests. There is a problem with use of our UK-CA (IGTF) certificates. There is a workaround in place but some more definitive solution is needed.
  • Euclid
    • An RT ticket has been created to track progress. There is a problem getting the Euclid environment in our Docker images and an issue of job sequencing.
  • Procedure Improvements:
    • The Grafana plot of non-LHC VO batch jobs was presented. This provides useful information. Some details of the layout to be tweaked.