Difference between revisions of "Tier1 Operations Report 2016-05-25"

From GridPP Wiki
Jump to: navigation, search
()
()
 
(5 intermediate revisions by one user not shown)
Line 11: Line 11:
 
* On Friday (20th) Atlas put batch queues offline. This can be seen in the HammerCloud results below. Problem traced to Atlas Castor instance which was fixed during the day.
 
* On Friday (20th) Atlas put batch queues offline. This can be seen in the HammerCloud results below. Problem traced to Atlas Castor instance which was fixed during the day.
 
* Also on Friday (20th) there was a problem seen on a disk servers in CMSDisk that was running out of xroot connections. This limit was increased although that change was not picked up by all servers in the service class. There was a further problem with one disk server over the weekend which was resolved on Monday by forcing the recent change to be picked up by all (CMSDisk) disk servers.
 
* Also on Friday (20th) there was a problem seen on a disk servers in CMSDisk that was running out of xroot connections. This limit was increased although that change was not picked up by all servers in the service class. There was a further problem with one disk server over the weekend which was resolved on Monday by forcing the recent change to be picked up by all (CMSDisk) disk servers.
* Atlas put our batch queues offline yesterday (24th May). This was chased to an error in the new code that creates the 'machine job features'. This was disabled this morning.
+
* Atlas put our batch queues offline yesterday (24th May). This was chased to an error in the new code that creates the 'machine job features'. This piece of code was disabled this morning and the problem fixed.
* This morning we have a problem with the Tape Library control software that is crashing. This means tape access is affected and a call is open with Oracle.
+
* This morning (25th) we have a problem with the Tape Library control software that is crashing. Investigations are underway. Tape access is affected and a call has been opened with Oracle.
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- ***********End Review of Issues during last week*********** ----->
 
<!-- *********************************************************** ----->
 
<!-- *********************************************************** ----->
Line 23: Line 23:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Resolved Disk Server Issues
 
|}
 
|}
* None
+
* GDSS635 (AtlasTape - D0T1) which had failed in the early hours of the 20th April was returned to service on Monday afternoon (23rd May). All its disks have been replaced and it has re-run the acceptance tests for a week. However, the next day, Tuesday (24th), the server was found not to be working correctly. It was taken out of production for a software problem to be identified and fixed. It was returned to production this morning.
 +
* GDSS619 (GenTape - D0T1) failed on 26th April. As for GDSS635 (above) all its disks have been replaced and it has re-run the acceptance tests for a week. It was also returned to service on Monday (23rd May).
 +
* GDSS664 (AtlasTape - D0T1) failed on Monday morning, 16th May. A verification of the RAID array found a faulty disk which has been replaced. It was returned to service last Wednesday (18th May).
 +
* GDSS727 (CMSDIsk reported 'FSProbe' problems on Thursday (19th May) and was taken out or service. Following the replacement of two disk drives it was returned to service on Sunday evening (22nd May).
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***********End Resolved Disk Server Issues*********** ----->
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 46: Line 49:
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues
 
| style="background-color: #f8d6a9; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Ongoing Disk Server Issues
 
|}
 
|}
* The first two disk servers listed below have failed more than once and are being put through a week long acceptance test.
+
* None.
* GDSS635 (AtlasTape - D0T1) failed in the early hours of the 20th April.
+
* GDSS619 (GenTape - D0T1) failed on 26th April only a few hours after being put back into service after a previous failure.
+
* GDSS664 (AtlasTape - D0T1) failed on Monday morning, 16th May. A verification of the RAID array found a faulty disk which has been replaced. Expected back in production later today.
+
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ***************End Ongoing Disk Server Issues**************** ----->
 
<!-- ************************************************************* ----->
 
<!-- ************************************************************* ----->
Line 60: Line 60:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made since the last meeting.
 
|}
 
|}
* The migration of Atlas data from "C" to "D" tapes is continuing.
+
* The migration of Atlas data from "C" to "D" tapes is ongoing.
 
* Write access to GenScratch has been stopped.
 
* Write access to GenScratch has been stopped.
* One (of two) new batches of Worker Nodes has bee put into production on Monday/Tuesday of this week.
+
* One (of two) new batches of Worker Nodes has been put into production on Monday/Tuesday of this week.
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- *************End Notable Changes made this last week************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->

Latest revision as of 10:29, 25 May 2016

RAL Tier1 Operations Report for 25th May 2016

Review of Issues during the week 18th to 25th May 2016.
  • On Friday (20th) Atlas put batch queues offline. This can be seen in the HammerCloud results below. Problem traced to Atlas Castor instance which was fixed during the day.
  • Also on Friday (20th) there was a problem seen on a disk servers in CMSDisk that was running out of xroot connections. This limit was increased although that change was not picked up by all servers in the service class. There was a further problem with one disk server over the weekend which was resolved on Monday by forcing the recent change to be picked up by all (CMSDisk) disk servers.
  • Atlas put our batch queues offline yesterday (24th May). This was chased to an error in the new code that creates the 'machine job features'. This piece of code was disabled this morning and the problem fixed.
  • This morning (25th) we have a problem with the Tape Library control software that is crashing. Investigations are underway. Tape access is affected and a call has been opened with Oracle.
Resolved Disk Server Issues
  • GDSS635 (AtlasTape - D0T1) which had failed in the early hours of the 20th April was returned to service on Monday afternoon (23rd May). All its disks have been replaced and it has re-run the acceptance tests for a week. However, the next day, Tuesday (24th), the server was found not to be working correctly. It was taken out of production for a software problem to be identified and fixed. It was returned to production this morning.
  • GDSS619 (GenTape - D0T1) failed on 26th April. As for GDSS635 (above) all its disks have been replaced and it has re-run the acceptance tests for a week. It was also returned to service on Monday (23rd May).
  • GDSS664 (AtlasTape - D0T1) failed on Monday morning, 16th May. A verification of the RAID array found a faulty disk which has been replaced. It was returned to service last Wednesday (18th May).
  • GDSS727 (CMSDIsk reported 'FSProbe' problems on Thursday (19th May) and was taken out or service. Following the replacement of two disk drives it was returned to service on Sunday evening (22nd May).
Current operational status and issues
  • There is a problem seen by LHCb of a low but persistent rate of failure when copying the results of batch jobs to Castor. There is also a further problem that sometimes occurs when these (failed) writes are attempted to storage at other sites. A recent modification has improved, but not completed fixed this.
  • The intermittent, low-level, load-related packet loss seen over external connections is still being tracked. Likewise we have been working to understand some remaining low level of packet loss seen within a part of our Tier1 network.
Ongoing Disk Server Issues
  • None.
Notable Changes made since the last meeting.
  • The migration of Atlas data from "C" to "D" tapes is ongoing.
  • Write access to GenScratch has been stopped.
  • One (of two) new batches of Worker Nodes has been put into production on Monday/Tuesday of this week.
Declared in the GOC DB
Service Scheduled? Outage/At Risk Start End Duration Reason
lcgwms06.gridpp.rl.ac.uk, SCHEDULED OUTAGE 01/06/2016 11:00 30/06/2016 11:00 29 days, Server lcgwms06.gridpp.rl.ac.uk Decommissioning
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.
  • The Castor 2.1.15 update is pending. Testing has shown a database related performance issue which is being followed up. We await successful resolution of that problem and completion of testing before scheduling. Following advice from the developers we will not upgrade the SRMs before the Castor 2.1.15 upgrade.
  • Decommissioning of "GEN Scratch" storage in Castor. (Formally announced by EGI broadcast). Write access to this area has now been stopped in preparation for completely stopping access on the 20th June.
  • Decommissioning of lcgwms06. This will leave two WMS systems remaining in service.
  • The HAProxy Load Balancer will be added in front of the Production FTS3 Service.

Listing by category:

  • Databases:
    • Switch LFC/3D to new Database Infrastructure.
  • Castor:
    • Update SRMs to new version (includes updating to SL6).
    • Update to Castor version 2.1.15.
    • Migration of data from T10KC to T10KD tapes (Affects Atlas & LHCb data).
  • Networking:
    • Replace the UKLight Router. Then upgrade the 'bypass' link to the RAL border routers to 2*40Gbit.
  • Fabric
    • Firmware updates on remaining EMC disk arrays (Castor, LFC)
  • Grid Services
    • Once the use of the Load Balancer (HAProxy) has been proven for the test FTS service it will be extended to other services.
Entries in GOC DB starting since the last report.
  • None
Open GGUS Tickets (Snapshot during morning of meeting)
GGUS ID Level Urgency State Creation Last Update VO Subject
121687 Green Less Urgent On Hold 2016-05-20 2016-05-23 packet loss problems seen on RAL-LCG perfsonar
120954 Green Less Urgent In Progress 2016-04-21 2016-05-24 LHCb SRM endpoint entry in GOCDB
120920 Green Less Urgent In Progress 2016-04-19 2016-05-06 SNO+ XRootD issues at RAL
120810 Yellow Urgent In Progress 2016-04-13 2016-05-24 Biomed Decommissioning of SE srm-biomed.gridpp.rl.ac.uk - forbid write access for biomed users
120350 Green Less Urgent In Progress 2016-03-22 2016-05-06 LSST Enable LSST at RAL
119841 Amber Less Urgent On Hold 2016-03-01 2016-04-26 LHCb HTTP support for lcgcadm04.gridpp.rl.ac.uk
117683 Yellow Less Urgent On Hold 2015-11-18 2016-04-05 CASTOR at RAL not publishing GLUE 2
Availability Report

Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 729); CMS HC = CMS HammerCloud

Day OPS Alice Atlas CMS LHCb Atlas HC CMS HC Comment
18/05/16 100 100 100 100 100 100 100
19/05/16 100 100 100 100 100 100 100
20/05/16 100 100 100 97 100 86 76 Problem with test on all CEs. Looks like a CMS problem.
21/05/16 100 100 28 98 100 99 90 Atlas: Central monitoring problem; CMS: Single SRM test failure ("Input/output error")
22/05/16 100 100 76 100 100 100 94 Atlas: Central monitoring problem
23/05/16 100 100 100 98 100 100 100 Single SRM Test failure: User timeout.
24/05/16 100 100 100 100 96 98 100 Single SRM Test failure: (No such file or directory)