Latest revision as of 12:19, 24 July 2013

RAL Tier1 Operations Report for 24th July 2013

• On Thursday (18th) the main RAL link to Janet was failed over to the alternative route (via London) when one of the multiple connections to Reading failed. This was transparent to us.
• On Thursday (18th) a configuration error affecting all the CEs caused batch problems for a couple of hours until noticed and corrected.
• Yesterday (Tuesday 23rd) the primary OPN link to CERN failed and we switched over to the backup route. We ran on the backup link for around 7 hours until the problem (JANET ticket reports a broken fibre) was fixed.

A post mortem report of the Atlas Castor outage on 28-30 June has been prepared and can be seen at:

 https://www.gridpp.ac.uk/wiki/RAL_Tier1_Incident_20130628_Atlas_Castor_Outage

• GDSS664 (AtlasDataDisk, D1T0) failed on 11th July. Following a period when the server was down, which lasted until Tuesday 16th July, it has now been completely drained. All files that were on the server are available to Atlas. The server itself is undergoing hardware checks before being returned to service.

• There has been a problem for a few weeks with the batch server and investigations are continuing. The problem started at the same time as the batch server was updated. Although this update was rolled back the problem has remained.
• The uplink to the UKLight Router is running on a single 10Gbit link, rather than a pair of such links.
• The problem LHCb jobs failing due to long job set-up times is still under investigation. The recent updates to the CVMFS clients to v2.1.12 is promising.
• The testing of FTS3 is continuing and the service is being put on a more 'production' footing. (This runs in parallel with our existing FTS2 service).
• We are participating in xrootd federated access tests for Atlas.
• Testing is ongoing with the proposed new batch system (ARC-CEs, Condor, SL6). Atlas and CMS running work through this. ALICE, LHCb & H1 being brought on-board with the testing.

• None

• CMS & LHCb Castor instance (stager) were upgraded to version 2.1.13-9 yesterday (Tuesday 23rd).

• None

• Update the remaining Castor stager GEN on Tuesday 30th July.
• The SL6 and "Whole Node" queues on the production batch service will be terminated. Multi-core jobs and those requiring SL6 can be run on the test Condor batch system.
• Wednesday 24th July: Transition of Thames Valley Network to Janet 6.
• Re-establishing the paired (2*10Gbit) link to the UKLight router.

Listing by category:

• Databases:
  • Switch LFC/FTS/3D to new Database Infrastructure.
• Castor:
  • Upgrade to version 2.1.13 (ongoing)
• Networking:
  • Single link to UKLight Router to be restored as paired (2*10Gbit) link.
  • Update core Tier1 network and change connection to site and OPN including:
    • Install new Routing layer for Tier1
    • Change the way the Tier1 connects to the RAL network.
    • These changes will lead to the removal of the UKLight Router.
• Grid Services
  • Testing of alternative batch systems (SLURM, Condor) along with ARC-CEs and SL6 Worker Nodes.
• Fabric
  • One of the disk arrays hosting the FTS, LFC & Atlas 3D databases is showing a fault and an intervention is required.
• Infrastructure:
  • A 2-day maintenance is being planned sometime in October or November for the following. This is expected to require around a half day outage of power to the UPS room with castor & Batch down for the remaining 1.5 days as equipment is switched off in rotation for the tests.
    • Intervention required on the "Essential Power Board" & Remedial work on three (out of four) transformers.
    • Remedial work on the BMS (Building Management System) due to one its three modules being faulty.
    • Electrical safety check. This will require significant (most likely 2 days) downtime during which time the above infrastructure issues will also be addressed.

There were no unscheduled entries in the GOC DB during this last week.