Tier1 Operations Report 2012-07-11

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 11th July 2012

Review of Issues during the week 4th to 11th July 2012
  • There was a problem with the database behind the Atlas/Gen SRMs for around an hour during the afternoon yesterday (Tuesday 10th July) when some logging caused a disk to fill.
Resolved Disk Server Issues
  • None
Current operational status and issues
  • On 12th/13th June the first stage of switching ready for the work on the main site power supply took place. The work on the two transformers is expected to take until 18th December and involves powering off one half of the resilient supply for 3 months while being overhauled, then repeat with the other half.
Ongoing Disk Server Issues
  • GDSS607 (LHCbDst - D1T0) has been out of service for some time. It is being swapped for a different server which is being acceptance tested ahead of deployment.
  • GDSS447 (Atlas DataDisk - d1T0) failed with multiple disk failures last night (10-11 July). The disk server is rebuilding its raid array. There is a significant risk of data loss from this incident, but it is too early to say yet.
Notable Changes made this last week
  • DataGuard was re-instated for the Castor Oracle databases on Thursday (5th July.)
  • The Castor Information Provider (CIP) was upgraded on Monday 9th July. This resolves an issue of disk space reporting.
  • This morning (11th July) CMS batch work was switched to use CVMFS.
Declared in the GOC DB
  • None
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Databases:
    • Switch LFC/FTS/3D to new Database Infrastructure.
  • Castor:
    • Upgrade to version 2.1.12.
  • Networking:
    • Install new Routing layer for Tier1 and update the way the Tier1 connects to the RAL network. (Plan to co-locate with replacement of UKlight network).
    • Update Spine layer for Tier1 network.
    • Replacement of UKLight Router.
    • Addition of caching DNSs into the Tier1 network.
  • Grid Services:
    • The FTS Agents are being progressively moved to virtual machines.
    • Updates of Grid Services as appropriate. (Services now on EMI/UMD versions unless there is a specific reason not.)


Entries in GOC DB starting between 4th and 11th July 2012

There were no GOC DB entries for this period.

Open GGUS Tickets
GGUS ID Level Urgency State Creation Last Update VO Subject
68853 Red Less Urgent On hold 2011-03-22 2012-06-25 N/A Retirenment of SL4 and 32bit DPM Head nodes and Servers
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2012-07-11&oldid=3249"