Revision as of 09:35, 9 April 2014

RAL Tier1 Operations Report for 9th April 2014

• There was a short (around 5 minute) break in external connectivity to the Tier1 during the morning of Thursday 20th March and again a similar event the following morning.
• There was a failover of an Atlas Castor Database early evening on Tuesday 25th March. The failover triggered a call-out and the database was put back onto its allocated node. The cause is a bug that has been reported to Oracle.
• On Friday, 28th March, we were not running some of the CE SUM tests in a timely manner. It was found that owing to a separate change in the Condor configuration we were no longer prioritising the test jobs. This was fixed.

• Last Wednesday (2nd April) GDSS239 (Atlas HotDisk) crashed. As AtlasHotDisk is being merged into another Space Token and there should be multiple copied of files on each server spread across the servers in AtalsHotDisk it was decided to withdraw teh server from use rather than invest time investigating. (No of unique files found?)
• In the early hours of Sunday 6th April GDSS600 (AtlasDataDisk - D1T0) failed. Multiple disk failures were being reported by teh disk controller. The system was returned to production yesterday evening (8th April) and is being drained. It will be decommissioned after files ahve been copied off.

• There have been problems with the CMS Castor instance in recent weeks. These are triggered by high load. Work is underway to alleviate these problems, in particular servers with faster network connections will be moved into the disk cache in front of CMS_Tape when they become available.
• The Castor Team are now able to reproduce the intermittent failures of Castor access via the SRM that has been reported in recent weeks. Understanding of the problem is significantly adcanced and further investigations are ongoing using the Castor Preprod instance. Ideas for a workaround are being developed.
• As reported before, working with Atlas the file deletion rate was somewhat improved. However, there is still a problem that needs to be understood.
• Around 50 files in tape backed service classes (mainly in GEN) have been found not to have migrated to tape. This is under investigation. The cause for some of these is understood (a bad tape at time of migration).
• Problems with the infrastructure used to host many of our non-Catsor services have largely been worked around, although not yet fixed. Some additional migrations of VMs has been necessary.

• None.

• The rollout of of WNs updated to the EMI-3 version has been completed.
• The EMI3 Argus server is now in use everyweher in the batch farm.
• Two of the CV2013 disk servers (120TB each) have been added to LHCbDst. A further 9 are being added today. Three further servers are in CMS non-prod awaiting being moved into production imminently. ?? Update this.

Listing by category:

• Databases:
  • Switch LFC/FTS/3D to new Database Infrastructure.
• Castor:
  • Castor 2.1.14 testing is largely complete. (A non-Tier1 production Castor instance was successfully upgraded yesterday, 1st April.) We are starting to look at possible dates for rolling this out (probably around May).
• Networking:
  • Update core Tier1 network and change connection to site and OPN including:
    • Install new Routing layer for Tier1 & change the way the Tier1 connects to the RAL network. (Scheduled for 29th April)
    • These changes will lead to the removal of the UKLight Router.
• Fabric
  • We are phasing out the use of the software server used by the small VOs.
  • Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
  • There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud