Tier1 Operations Report 2013-05-01

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 1st May 2013

Review of Issues during the week 24th April to 1st May 2013.
  • A quiet week. There was a short break when network router 'A' was rebooted at 10:30 on Friday (26th April).
  • The Tier1 uplink was saturated outbound from around Tuesday (23rd April) until Monday (29th April). This was traced to a CMS test.
  • One file was declared lost to Atlas on Friday (26th April). The file had been written early in February but was found to be corrupt (bad checksum) when being read by Atlas. Checks on the disk server failed to find any other problems.
Resolved Disk Server Issues
  • None.
Current operational status and issues
  • The uplink to the UKLight Router is running on a single 10Gbit link, rather than a pair of such links.
  • The problem LHCb and Atlas jobs failing due to long job set-up times remains and investigations continue.
  • The testing of FTS3 is continuing. (This runs in parallel with our existing FTS2 service).
  • We are participating in xrootd federated access tests for Atlas.
  • Test batch queue with five SL6/EMI-2 worker nodes and own CE in place.
  • There is an outstanding problem (and GGUS ticket) affecting the certificate on the MyProxy server.
Ongoing Disk Server Issues
  • None.
Notable Changes made this last week
  • A fix to the Castor diskmanagerd process to resolve the intermittent problem of timeouts (seen in SUM tests) was rolled out to the CMS disk servers on Thursday (25th April) and to all remaining disk servers on Monday (29th).
  • On Friday (26th) lcgce12 (CE for SL6 worker nodes) was added to the siteBDII configuration.
  • On Monday (29th April) seven disk servers from the second batch of 2012 orders were put into production in CMSDisk.
  • This morning (1st May) Oracle PSU patches were applied to the main Castor databases.
Declared in the GOC DB
  • None
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • The Castor production and standby databases will be switched over (requiring Castor stop). Scheduling within next fortnight.
  • Re-establishing the paired (2*10Gbit) link to the UKLight router. (Aiming to do in next weeks).
  • The blocking issue regarding the Castor 2.1.13 upgrade has been resolved and the scheduling of this upgrade will proceed.
  • A program of updating the disk controller firmware in the 2011 Clustervision batch of disk servers is ongoing. (Alice disk servers in this batch remain to be done).

Listing by category:

  • Databases:
    • Switch LFC/FTS/3D to new Database Infrastructure.
  • Castor:
    • Upgrade to version 2.1.13
  • Networking:
    • Single link to UKLight Router to be restored as paired (2*10Gbit) link.
    • Update core Tier1 network and change connection to site and OPN including:
      • Install new Routing layer for Tier1
      • Change the way the Tier1 connects to the RAL network.
      • These changes will lead to the removal of the UKLight Router.
    • Addition of caching DNSs into the Tier1 network.
  • Grid Services
    • Testing of alternative batch systems (e.g. SLURM).
    • Upgrade of one remaining EMI-1 component (UI) being planned.
  • Fabric
    • One of the disk arrays hosting the FTS, LFC & Atlas 3D databases is showing a fault and an intervention is required.
  • Infrastructure:
    • Intervention required on the "Essential Power Board" & Remedial work on three (out of four) transformers.
    • Remedial work on the BMS (Building Management System) due to one its three modules being faulty.
    • Electrical safety check. This will require significant (most likely 2 days) downtime during which time the above infrastructure issues will also be addressed.
Entries in GOC DB starting between 24th April and 1st May 2013.

There were no unscheduled outages during the last week.

Service Scheduled? Outage/At Risk Start End Duration Reason
All Castor (srm-alice, srm-atlas, srm-biomed, srm-cms, srm-dteam, srm-hone, srm-ilc, srm-lhcb, srm-mice, srm-minos, srm-na62, srm-snoplus, srm-superb, srm-t2k) SCHEDULED WARNING 01/05/2013 08:00 01/05/2013 12:00 4 hours At Risk on storage (Castor) services during application of Oracle patches to back end database systems.
lcgfts.gridpp.rl.ac.uk, lfc.gridpp.rl.ac.uk SCHEDULED WARNING 24/04/2013 09:00 24/04/2013 13:00 4 hours Warning during application of Oracle paches to back-end databases behind FTS, LFC and Atlas 3D systems.


Open GGUS Tickets (Snapshot at time of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
93149 Red Less Urgent On Hold 2013-04-05 2013-04-08 Atlas RAL-LCG2: jobs failing with " cmtside command was timed out"
92266 Red Less Urgent Waiting for Reply 2013-03-06 2013-04-16 Certificate for RAL myproxy server
91658 Red Less Urgent On Hold 2013-02-20 2013-04-03 LFC webdav support
86152 Red Less Urgent On Hold 2012-09-17 2013-03-19 correlated packet-loss on perfsonar host
Availability Report
Day OPS Alice Atlas CMS LHCb Comment
24/04/13 100 100 100 100 100
25/04/13 100 100 99.2 100 100 Single SRM test failure "could not open connection to srm-atlas.gridpp.rl.ac.uk"
26/04/13 100 100 100 100 100
27/04/13 100 100 100 100 100
28/04/13 100 100 100 100 100
29/04/13 100 100 100 100 100
30/04/13 100 100 100 100 100
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2013-05-01&oldid=3532"