Difference between revisions of "Tier1 Operations Report 2014-09-24"
From GridPP Wiki
(→) |
(→) |
||
(6 intermediate revisions by one user not shown) | |||
Line 9: | Line 9: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 17th to 24th September 2014. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 17th to 24th September 2014. | ||
|} | |} | ||
− | * On | + | * On Thursday morning (around 07:40) there was a crash of the Atlas SRM database. The database immediately restarted on a different RAC node, and has subsequently been put back. This is a repeat of a known and reported Oracle bug. |
− | + | * There was some very high load on a number of disk servers in AtlasDataDisk, particularly during the latter part of last week. Local measures to improve the problem didn't help. Atlas are aware of issues caused by a partiular workflow. | |
− | * | + | * Yesterday, Tues 23rd Sep, the Atlas Frontier squids were switched to use a new database system ("Cronos"). However, during the evening problems were found with the Frontier squids accessing the Cronos database and it was switched back to the former database ("Ogma"). The Cronos Atlas 3D database updates from CERN via Oracle "Goldengate" which supersedes the Oracle "Streams" method used to update Ogma. |
<!-- ***********End Review of Issues during last week*********** -----> | <!-- ***********End Review of Issues during last week*********** -----> | ||
<!-- *********************************************************** -----> | <!-- *********************************************************** -----> | ||
Line 34: | Line 34: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues | ||
|} | |} | ||
− | * | + | * We are investigating why the Atlas Frontier systems had problems using the new Cronos database. |
<!-- ***********End Current operational status and issues*********** -----> | <!-- ***********End Current operational status and issues*********** -----> | ||
<!-- *************************************************************** -----> | <!-- *************************************************************** -----> | ||
Line 56: | Line 56: | ||
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week. | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notable Changes made this last week. | ||
|} | |} | ||
− | * VO | + | * VO Pheno have been enabled to use Castor. |
+ | * The production FTS3 instance upgraded to 3.2.27 | ||
<!-- *************End Notable Changes made this last week************** -----> | <!-- *************End Notable Changes made this last week************** -----> | ||
<!-- ****************************************************************** -----> | <!-- ****************************************************************** -----> | ||
Line 67: | Line 68: | ||
| style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB | | style="background-color: #d8e8ff; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Declared in the GOC DB | ||
|} | |} | ||
− | + | ||
+ | {| border=1 align=center | ||
+ | |- bgcolor="#7c8aaf" | ||
+ | ! Service | ||
+ | ! Scheduled? | ||
+ | ! Outage/At Risk | ||
+ | ! Start | ||
+ | ! End | ||
+ | ! Duration | ||
+ | ! Reason | ||
+ | |- | ||
+ | | arc-ce02, arc-ce03, arc-ce04 | ||
+ | | SCHEDULED | ||
+ | | WARNING | ||
+ | | 09/10/2014 10:00 | ||
+ | | 09/10/2014 11:00 | ||
+ | | 1 hour | ||
+ | | Upgrade of ARC CEs to version 4.2.0 | ||
+ | |- | ||
+ | | arc-ce01 | ||
+ | | SCHEDULED | ||
+ | | WARNING | ||
+ | | 07/10/2014 10:00 | ||
+ | | 07/10/2014 11:00 | ||
+ | | 1 hour | ||
+ | | Upgrade of ARC CE to version 4.2.0 | ||
+ | |} | ||
+ | |||
<!-- **********************End GOC DB Entries************************** -----> | <!-- **********************End GOC DB Entries************************** -----> | ||
<!-- ****************************************************************** -----> | <!-- ****************************************************************** -----> | ||
Line 84: | Line 112: | ||
* Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday | * Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday | ||
30th September. | 30th September. | ||
− | |||
'''Listing by category:''' | '''Listing by category:''' | ||
* Databases: | * Databases: | ||
Line 111: | Line 138: | ||
| style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting between the 17th and 24th September 2014. | | style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting between the 17th and 24th September 2014. | ||
|} | |} | ||
− | + | {| border=1 align=center | |
+ | |- bgcolor="#7c8aaf" | ||
+ | ! Service | ||
+ | ! Scheduled? | ||
+ | ! Outage/At Risk | ||
+ | ! Start | ||
+ | ! End | ||
+ | ! Duration | ||
+ | ! Reason | ||
+ | |- | ||
+ | | lcgfts3.gridpp.rl.ac.uk, | ||
+ | | SCHEDULED | ||
+ | | WARNING | ||
+ | | 23/09/2014 10:00 | ||
+ | | 23/09/2014 12:00 | ||
+ | | 2 hours | ||
+ | | Service At Risk during upgrade to FTS3 version 3.2.27. | ||
+ | |} | ||
<!-- **********************End GOC DB Entries************************** -----> | <!-- **********************End GOC DB Entries************************** -----> | ||
<!-- ****************************************************************** -----> | <!-- ****************************************************************** -----> |
Latest revision as of 10:59, 24 September 2014
RAL Tier1 Operations Report for 24th September 2014
Review of Issues during the week 17th to 24th September 2014. |
- On Thursday morning (around 07:40) there was a crash of the Atlas SRM database. The database immediately restarted on a different RAC node, and has subsequently been put back. This is a repeat of a known and reported Oracle bug.
- There was some very high load on a number of disk servers in AtlasDataDisk, particularly during the latter part of last week. Local measures to improve the problem didn't help. Atlas are aware of issues caused by a partiular workflow.
- Yesterday, Tues 23rd Sep, the Atlas Frontier squids were switched to use a new database system ("Cronos"). However, during the evening problems were found with the Frontier squids accessing the Cronos database and it was switched back to the former database ("Ogma"). The Cronos Atlas 3D database updates from CERN via Oracle "Goldengate" which supersedes the Oracle "Streams" method used to update Ogma.
Resolved Disk Server Issues |
- GDSS763 (AtlasDataDisk - D1T0) failed in the early morning of Thursday 18th Sep. It was restarted and tested but no fault found. The system was returned to service early the following afternoon (19th Sep).
- GDSS720 (AtlasDataDisk - D1T0) failed on Sunday afternoon, 21st Sep. It was restarted and tested but no fault found. The system was returned to service at the end of Monday afternoon (22nd Sep).
Current operational status and issues |
- We are investigating why the Atlas Frontier systems had problems using the new Cronos database.
Ongoing Disk Server Issues |
- None.
Notable Changes made this last week. |
- VO Pheno have been enabled to use Castor.
- The production FTS3 instance upgraded to 3.2.27
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
arc-ce02, arc-ce03, arc-ce04 | SCHEDULED | WARNING | 09/10/2014 10:00 | 09/10/2014 11:00 | 1 hour | Upgrade of ARC CEs to version 4.2.0 |
arc-ce01 | SCHEDULED | WARNING | 07/10/2014 10:00 | 07/10/2014 11:00 | 1 hour | Upgrade of ARC CE to version 4.2.0 |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
- The rollout of the RIP protocol to the Tier1 routers still has to be completed.
- Access to the Cream CEs will be withdrawn apart from leaving access for ALICE. The proposed date for this is Tuesday
30th September. Listing by category:
- Databases:
- Apply latest Oracle patches (PSU) to the production database systems (Castor, LFC).
- A new database (Oracle RAC) has been set-up to host the Atlas3D database. This is updated from CERN via Oracle GoldenGate.
- Switch LFC/3D to new Database Infrastructure.
- Castor:
- Update Castor headnodes to SL6.
- Fix discrepancies were found in some of the Castor database tables and columns. (The issue has no operational impact.)
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Enable the RIP protocol for updating routing tables on the Tier1 routers.
- Fabric
- Migration of data to new T10KD tapes. (Migration of CMS from 'B' to 'D' tapes underway; migration of GEN from 'A' to 'D' tapes to follow.)
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room (Expected first quarter 2015).
Entries in GOC DB starting between the 17th and 24th September 2014. |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
lcgfts3.gridpp.rl.ac.uk, | SCHEDULED | WARNING | 23/09/2014 10:00 | 23/09/2014 12:00 | 2 hours | Service At Risk during upgrade to FTS3 version 3.2.27. |
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
108546 | Green | Less Urgent | In Progress | 2014-09-16 | 2014-09-22 | Atlas | RAL-LCG2_HIMEM_SL6: production jobs failed |
107935 | Amber | Less Urgent | In Progress | 2014-08-27 | 2014-09-02 | Atlas | BDII vs SRM inconsistent storage capacity numbers |
107880 | Red | Less Urgent | In Progress | 2014-08-26 | 2014-09-02 | SNO+ | srmcp failure |
106324 | Red | Urgent | In Progress | 2014-06-18 | 2014-09-23 | CMS | pilots losing network connections at T1_UK_RAL |
Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | CMS HC | Comment |
---|---|---|---|---|---|---|---|---|
17/09/14 | 100 | 100 | 100 | 100 | 100 | 98 | 96 | |
18/09/14 | 100 | 100 | 100 | 100 | 100 | 99 | 99 | |
19/09/14 | 100 | 100 | 100 | 100 | 100 | 100 | 98 | |
20/09/14 | 100 | 100 | 100 | 100 | 100 | 96 | 97 | |
21/09/14 | 100 | 100 | 100 | 100 | 100 | 97 | 97 | |
22/09/14 | 100 | 100 | 100 | 100 | 100 | 95 | 98 | |
23/09/14 | 100 | 100 | 100 | 100 | 100 | 98 | 98 |