Difference between revisions of "Tier1 Operations Report 2017-07-19"

Revision as of 11:50, 19 July 2017

RAL Tier1 Operations Report for 19th July 2017

• There was a problem with the Atlas Castor instance for a few hours during the afternoon of Friday 14th July. A problem on the SRMs correlated with a spike in the SRM request rate.
• On Sunday (16th July) there was a problem with the CMS Castor instance. On-call staff responded. The back-end database reported locking sessions and hot-spotting of files was seen. The problem affected batch access to files as well. Since then there have still been some indication of CMS Castor problems (with 'unable to issue PrepareToPut request to Castor' errors) but at a much reduced rate.
• There is a power outage scheduled for the Atlas Building (R26) over the weekend of 29/30 July. Preparations are being made to remove any impact this may have on operational services.

• GDSS650 (LHCbUser - D1T0) failed on Sunday 16th July. The server was being drained. A disk was replaced and the server returned to service this morning (19th July).

• We are still seeing a rate of failures of the CMS SAM tests against the SRM. These are affecting our (CMS) availabilities. CMS are also looking at file access performance and have turned off "lazy-download". The CMS SRM SAM test success rate has improved since the Castor 2.1.16 upgrade on the 25th May, although is still not 100%. It is still planned to re-visit this issue now Castor has been upgraded.
• There is a problem on the site firewall which is causing problems for some specific data flows. Discussions have been taking place with the vendor. It is expected that this is having an effect on our data that flows through the firewall (such as to/from worker nodes).

• None

• CMS Multicore 550

• The number of placement groups in the Echo CEPH Atlas pool contiues to be increased. This is in preparation for the increase in storage capacity when new hardware is added.
• Security patching being carried out across systems.
• Updating of RAID card firmware in one batch of disk servers (OCF '14) is taking place.
• On Thursday 13th July the WAN tuning parameters were updated on CASTOR disk servers. This had been applied to some of the servers some months ago and this completed the roll-out of this change and standardized these parameters across the Castor disk servers.
• On Monday 17th July access to FTS3 via the SOAP interface was stopped by removing the fts3-prod-soap proxy from the load balancers.

• None

Pending - but not yet formally announced:

• Firmware updates in OCF 14 disk servers. These will be done next week (17-21 July).
• Upgrade the FTS3 service to a version that will no longer support the SOAP interface. The SOAP interface will be disabled on the 17th July.
• Increase the number of placement groups in the Atlas Echo CEPH pool. (Ongoing)

Listing by category:

• Castor:
  • Move to generic Castor headnodes.
  • Merge AtlasScratchDisk into larger Atlas disk pool.
• Echo:
  • Increase the number of placement groups in the Atlas Echo CEPH pool.
• Networking
  • Enable first services on production network with IPv6 now that the addressing scheme has been agreed. (Perfsonar already working over IPv6).
• Services
  • The production FTS needs updating. This will no longer support the SOAP interface. (The "test" FTS , used by Atlas, has already been upgraded.)

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 845); Atlas HC Echo = Atlas Echo (Template 841);CMS HC = CMS HammerCloud

• None yet