RAL Tier1 Incident 20160600 Tape Library Software Crashes

From GridPP Wiki
Revision as of 12:28, 20 June 2016 by Gareth Smith e6a7c1e977 (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

RAL-LCG2 Incident 20160600 Tape Library Software Crashest

Description:

The Control Software for the Tape Libraries (ACSLS) crashes/hangs frequently

Impact

Describe the type of impact. Include which services / VOs. How long they were impacted for and give the dates. If data loss ensure this is clearly flagged.


Timeline of the Incident

When What
17th February 2016 Upgrade all tape servers to 2.1.15-20
25th April 2016 Upgrade tape servers lcgcts{12,13} to 2.1.16-0 (7% of drives)
26th April 2016 Upgrade tape servers lcgcts{10,11,20,21,27} to 2.1.16-0 (31% of drives; lcgcts{20,21} have two drives attached)
17th May 2016 Upgrade all remaining tape servers to 2.1.16-0


Incident details

Put a reasonably detailed description of the incident here.


Analysis

This section to include a breakdown of what happened. Include any related issues.


Follow Up

This is what we used to call future mitigation. Include specific points to be done. It is not necessary to use the table below, but may be easier to do so.


Issue Response Done
Issue 1 Mitigation for issue 1. Done yes/no
Issue 2 Mitigation for issue 2. Done yes/no

Related issues

List any related issue and provide links if possible. If there are none then remove this section.


Reported by: Your Name at date/time

Summary Table

Start Date Date e.g. 20 July 2010
Impact Select one of: >80%, >50%, >20%, <20%
Duration of Outage Hours e.g. 3hours
Status select one from Draft, Open, Understood, Closed
Root Cause Select one from Unknown, Software Bug, Hardware, Configuration Error, Human Error, Network, User Load
Data Loss Yes/No