Difference between revisions of "Tier1 Operations Report 2014-07-09"
From GridPP Wiki
(Created page with "=RAL Tier1 Operations Report for 2nd July 2014= __NOTOC__ ====== ====== <!-- ************************************************************* -----> <!-- ***********Start Review...") |
|||
Line 1: | Line 1: | ||
− | =RAL Tier1 Operations Report for | + | =RAL Tier1 Operations Report for 9th July 2014= |
__NOTOC__ | __NOTOC__ | ||
====== ====== | ====== ====== | ||
Line 7: | Line 7: | ||
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | {| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | ||
|- | |- | ||
− | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week | + | | style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Review of Issues during the week 2nd to 9th July 2014. |
|} | |} | ||
* There were Atlas SRM test failures during last week which were traced to load on the Atlas Castor system during searches for dark data. Less intrusive ways of carrying out tasks such as this are being investigated. | * There were Atlas SRM test failures during last week which were traced to load on the Atlas Castor system during searches for dark data. Less intrusive ways of carrying out tasks such as this are being investigated. | ||
Line 120: | Line 120: | ||
{| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | {| width="100%" cellspacing="0" cellpadding="0" style="background-color: #ffffff; border: 1px solid silver; border-collapse: collapse; width: 100%; margin: 0 0 1em 0;" | ||
|- | |- | ||
− | | style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting between the | + | | style="background-color: #7c8aaf; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Entries in GOC DB starting between the 2nd and 9th July 2014. |
|} | |} | ||
{| border=1 align=center | {| border=1 align=center | ||
Line 178: | Line 178: | ||
|-style="background:#b7f1ce" | |-style="background:#b7f1ce" | ||
! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject | ! GGUS ID !! Level !! Urgency !! State !! Creation !! Last Update !! VO !! Subject | ||
+ | |- | ||
+ | | 106640 | ||
+ | | Green | ||
+ | | Less Urgent | ||
+ | | In Progress | ||
+ | | 2014-07-04 | ||
+ | | 2014-07-04 | ||
+ | | ILC | ||
+ | | Failure to submit jobs to RAL-LCG2 CEs | ||
+ | |- | ||
+ | | 106610 | ||
+ | | Green | ||
+ | | Less Urgent | ||
+ | | In Progress | ||
+ | | 2014-07-02 | ||
+ | | 2014-07-02 | ||
+ | | HyperK | ||
+ | | HyperK support | ||
|- | |- | ||
| 106480 | | 106480 | ||
Line 214: | Line 232: | ||
| | | | ||
| please check your Vidyo router firewall configuration | | please check your Vidyo router firewall configuration | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
|} | |} | ||
<!-- **********************End GGUS Tickets************************** -----> | <!-- **********************End GGUS Tickets************************** -----> | ||
Line 246: | Line 255: | ||
| 26/06/14 || 100 || 100 || style="background-color: lightgrey;" |90.6 || style="background-color: lightgrey;" |95.8 || style="background-color: lightgrey;" |92.6 || 90 || 100 || LHCb Castor Stager 2.1.14 upgrade; Atlas: Several SRM test failures; CMS: Single SRM Put test failure. | | 26/06/14 || 100 || 100 || style="background-color: lightgrey;" |90.6 || style="background-color: lightgrey;" |95.8 || style="background-color: lightgrey;" |92.6 || 90 || 100 || LHCb Castor Stager 2.1.14 upgrade; Atlas: Several SRM test failures; CMS: Single SRM Put test failure. | ||
|- | |- | ||
− | | | + | | 02/07/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || |
+ | |- | ||
+ | | 03/07/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || | ||
+ | |- | ||
+ | | 04/07/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || | ||
|- | |- | ||
− | | | + | | 05/07/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || |
|- | |- | ||
− | | | + | | 06/07/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || |
|- | |- | ||
− | | | + | | 07/07/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || |
|- | |- | ||
− | | | + | | 08/07/14 || 100 || 100 || 100 || 100 || 100 || 100 || 100 || |
|} | |} | ||
<!-- **********************End Availability Report************************** -----> | <!-- **********************End Availability Report************************** -----> | ||
<!-- *********************************************************************** -----> | <!-- *********************************************************************** -----> |
Revision as of 15:09, 4 July 2014
RAL Tier1 Operations Report for 9th July 2014
Review of Issues during the week 2nd to 9th July 2014. |
- There were Atlas SRM test failures during last week which were traced to load on the Atlas Castor system during searches for dark data. Less intrusive ways of carrying out tasks such as this are being investigated.
- There was a problem with one (of the three) Top-BDII systems which was unavailable for around 40 minutes from midday yesterday (1st July).
- A UPS/Generator load test was successfully carried out this morning (Wednesday 2nd July).
Resolved Disk Server Issues |
- None
Current operational status and issues |
- We continue to monitor closely the performance of xroot access to CMS Castor following the upgrade on the 17th June. Performance is generally good although the current CMS workflow causes some problems with hot files.
Ongoing Disk Server Issues |
- None
Notable Changes made this last week. |
- Thursday (26th June) LHCb Castor instance upgraded to version 2.1.14-13.
Declared in the GOC DB |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
srm-atlas.gridpp.rl.ac.uk, | SCHEDULED | OUTAGE | 08/07/2014 06:00 | 09/07/2014 12:00 | 1 day, 6 hours | Atlas Castor instance down for Castor 2.1.14 Stager Update |
Advanced warning for other interventions |
The following items are being discussed and are still to be formally scheduled and announced. |
- Date for the final Castor 2.1.14 stager upgrades of Atlas: Tuesday 8th July.
- We are planning the termination of the FTS2 service (proposed for 2nd September) now that almost all use is on FTS3.
Listing by category:
- Databases:
- Switch LFC/FTS/3D to new Database Infrastructure.
- Castor:
- The Castor 2.1.14 upgrade is underway.
- Networking:
- Move switches connecting the 2011 disk servers batches onto the Tier1 mesh network.
- Make routing changes to allow the removal of the UKLight Router.
- Fabric
- We are phasing out the use of the software server used by the small VOs.
- Firmware updates on remaining EMC disk arrays (Castor, FTS/LFC)
- There will be circuit testing of the remaining (i.e. non-UPS) circuits in the machine room during 2014.
Entries in GOC DB starting between the 2nd and 9th July 2014. |
Service | Scheduled? | Outage/At Risk | Start | End | Duration | Reason |
---|---|---|---|---|---|---|
Whole Site | SCHEDULED | WARNING | 02/07/2014 10:00 | 02/07/2014 11:00 | 1 hour | RAL Tier1 site in warning state due to UPS/generator test. |
srm-lhcb.gridpp.rl.ac.uk | SCHEDULED | OUTAGE | 26/06/2014 10:00 | 26/06/2014 11:19 | 1 hour and 19 minutes | LHCb Castor instance down for Castor 2.1.14 Stager Update |
srm-lhcb.gridpp.rl.ac.uk | UNSCHEDULED | OUTAGE | 26/06/2014 09:00 | 26/06/2014 10:00 | 1 hour | Starting LHCb Castor instance Castor 2.1.14 Stager Update one hour earlier. |
srm-alice.gridpp.rl.ac.uk | UNSCHEDULED | WARNING | 24/06/2014 17:00 | 25/06/2014 12:00 | 19 hours | Alice-disk xroot availability at risk following Castor upgrade. |
Open GGUS Tickets (Snapshot during morning of meeting) |
GGUS ID | Level | Urgency | State | Creation | Last Update | VO | Subject |
---|---|---|---|---|---|---|---|
106640 | Green | Less Urgent | In Progress | 2014-07-04 | 2014-07-04 | ILC | Failure to submit jobs to RAL-LCG2 CEs |
106610 | Green | Less Urgent | In Progress | 2014-07-02 | 2014-07-02 | HyperK | HyperK support |
106480 | Green | Less Urgent | Waiting Reply | 2014-06-25 | 2014-06-30 | dteam | Publishing meaningful Castor version |
106324 | Yellow | Urgent | In Progress | 2014-06-18 | 2014-07-01 | CMS | pilots losing network connections at T1_UK_RAL |
105571 | Red | Less Urgent | In Progress | 2014-05-21 | 2014-06-30 | LHCb | BDII and SRM publish inconsistent storage capacity numbers |
105405 | Red | Urgent | On Hold | 2014-05-14 | 2014-07-01 | please check your Vidyo router firewall configuration |
Availability Report |
Key: Atlas HC = Atlas HammerCloud (Queue ANALY_RAL_SL6, Template 508); CMS HC = CMS HammerCloud
Day | OPS | Alice | Atlas | CMS | LHCb | Atlas HC | CMS HC | Comment |
---|---|---|---|---|---|---|---|---|
25/06/14 | 100 | 100 | 94.8 | 100 | 100 | 96 | 98 | Several SUM test failures (Invalid Argument). |
26/06/14 | 100 | 100 | 90.6 | 95.8 | 92.6 | 90 | 100 | LHCb Castor Stager 2.1.14 upgrade; Atlas: Several SRM test failures; CMS: Single SRM Put test failure. |
02/07/14 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
03/07/14 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
04/07/14 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
05/07/14 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
06/07/14 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
07/07/14 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
08/07/14 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |